Tuesday, 15 May 2018

Transposing data using UNPIVOT

A couple of years ago I posted a method to remove nulls from a report using the Value Attribute Paris - Column template.

Here's an example of how we might utilise the region, within the breadcrumb region position.

Note - some values may have been adjusted from this screenshot for their protection.

Any nulls were shown as a tilde, then hunted down and eliminated with some jQuery that executes after refresh of the region, and/or on load of the page

Cool idea, can be improved.

And when the JavaScript was placed on one line, it seems so innocuous. It works, so what? What's the harm?
$('dd.t-AVPList-value').each(function(){if ($(this).text().indexOf('~')>0) $(this).hide().prev().hide()})

The trouble with my original method is that it's executing jQuery after the page is rendered. This represents extra work that could be eliminated. The same justification was present in Oracle Forms, as Post-Query triggers were not preferred.

And of course the applies to DML being applied to the database. Sure, you may need some conditional processing and apply a zillion single updates, or you could write some elegant SQL to do it within one update.

The other problem is that the Universal Theme responds to this change with some content movement that can frustrate the user - the body bubbles up to meet the removed content.

New and Improved Solution

Side-note: how can it be new and improved?

In this case I've been talking about columns from a relative simple query, as simple as selecting from scott.emp.

If we can transpose these results, like a magic wand you can do in Excel, then we could just use the Value Attribute Pairs - Row version of the template, and not worry about any jQuery that manipulates the page after it's rendered.

We tranpose columns by surrounding the existing query with a simple UNPIVOT.

select * from (
    select to_char(empno) empno
     ,ename
     ,job
     ,to_char(sal) sal
     ,to_char(comm) comm
    from scott.emp
    where empno = 7788
    --where empno = 7654
) 
unpivot 
(val
 for name in (
    (empno) as 'Emp No'
   ,(ename) as 'Name'
   ,(job) as 'Job'
   ,(sal) as 'Sal'
   ,(comm) as 'Commission'
 )
);

NAME       VAL                                     
---------- --------
Emp No     7788                                    
Name       SCOTT                                   
Job        ANALYST                                
Sal        3000   

4 rows selected.

... query executed with other empno

NAME       VAL                                     
---------- ---------
Emp No     7654                                    
Name       MARTIN                                  
Job        SALESMAN                                
Sal        1250                                    
Commission 1400

5 rows selected.

Notice the result using an emp with a commission shows more rows. The page will only render the data supplied so there no dynamic action required.

Default ordering seems to honour the order of elements in the FOR expression.
Datatypes of columns must match up, hence the to_char around the numeric columns.

Update - I saw Mike on the forums suggest that "transpose" implies a matrix, and offered a combined unpivot/pivot.

tl;dr steps

  • surround the existing query with unpivot
  • add any datatype conversions necessary to make columns the same
  • change report region from pairs with column to pairs with row
  • remove declarative column ordering
  • remove the dynamic action that finds the tildes

If you want to modify how something presents itself on an APEX page, there are other options to explore before jQuery
  • Template options
  • Conditional SQL, manifesting as HTML expression in a column
  • CSS solutions trump JavaScript.
Simplify, man.

Monday, 14 May 2018

Filtering outliers from Oracle APEX activity logs

Last year I described a simple test case that described how to remove outliers from a fictional dataset using the STDDEV() analytical function .
http://www.grassroots-oracle.com/2017/06/removing-outliers-using-stddev.html

I want to follow this up with a practical case using one of my favourite data sets - the apex_workspace_activity_logs that record who opened what page, in what context, and how long it took to generate.

I've been keeping an eye on the performance of a particular page, after making a few performance adjustments to some conditions. Unfortunately, we had an unrelated anomaly that pushed average pages times quite high for a short period. Needless to say, this set of outliers transformed my beautiful performance indicating lines to a boxy bell curve.

Oracle APEX page performance data with extreme outlier

A great feature with Oracle JET is the ability to hide certain series, on click within the legend.
In this case I just wanted to ignore the MAX line for this post, which in this chart forms the secondary y-axis.

OracleJET Region Attributes - rescale
Our clients really enjoy this particular feature (so do I), so thanks to the JET team for building such a device, and the APEX team for integrating it.

This graph shows results where I modified the query to filter the outliers, on demand.

Performance graph with outlier removed

Looks like the adjustments to the conditions worked! The trend is downwards.

I tried a few variations to control the switch, but this seemed to perform the most predictably, although I'm not happy with the hardcoded number.

select [aggregate stuff]
from (
 select [all columns]
,case when :P23_IGNORE_OUTLIERS = 'Y' then
  -- only bother calculating when filtering them out
  stddev(elapsed_time) over (order by null)
else
  9999999
end as the_stddev
from [activity logs]
where [time/page is desired]
)
-- only when elapsed time less than 2 standard deviations gets 95% of your data
where (elapsed_time < 2*the_stddev )
I have a generic example of this on livesql.oracle.com I pay attention to these aggregates for our performance reports
  • Median - what most users are experiencing
  • Average - a typical user experience, influenced by extremes
  • Moving average - general trend of visits, spread over a few days. An attempt to normalise local events
  • Max - what's the worst some people are experiencing?
Here are some other activity log queries you may find interesting.

Happy graphing!

Tuesday, 24 April 2018

Logger Snippets for VS Code

Snippets are a common feature in text editors, allowing you to essentially paste in a snippet of code from some library somewhere.

Snippets appear for selection as you start typing, and the editor will paste in the entire content of the snippet.

Snippets in VS Code

VS code snippets also have the ability to focus the cursor, ready for typing.

It's very closely related to code completion concepts, applied in VS Code with Intellisense. Vito on Twitter suggested perhaps these should be incorporated into intellisense instead of snippets. I think perhaps that might be a personal choice.

I found following the guide useful in getting started
https://code.visualstudio.com/docs/editor/userdefinedsnippets

And I thought I'd start with the Logger template, since I've finally converted.

I quickly ended up in the deep end, but swimming well. In a few strokes, I can add this entire block of text, with the three instances of the procedure name highlighted, ready to type over. That's pretty cool.


This took 5 keystrokes

I figured a few variations of the entire text would be useful, so now as I type "log", I can tab, then start typing my log text.


 The first few commands at the start of a procedure form another snippet, with the parameter name ready to type.

If I'm going to logger, I'm saving some typing...

I extended this to include two parameters, and as you tab out of the first, the second two 'tabstops' are highlighted.

With all the magic going on by Adrian Png et al, I was a little surprised someone hadn't already published something for Logger.
If you're keen, here's my snippets so far. I haven't really thought too hard about the names yet, they're pretty much as I've smashed them out.
Others on Twitter suggested they might send through a few examples they use, I was going to wait for some more inspiration.

{
 // Place your global snippets here. Each snippet is defined under a snippet name and has a scope, prefix, body and
 // description. Add comma separated ids of the languages where the snippet is applicable in the scope field. If scope
 // is left empty or omitted, the snippet gets applied to all languages. The prefix is what is
 // used to trigger the snippet and the body will be expanded and inserted. Possible variables are:
 // $1, $2 for tab stops, $0 for the final cursor position, and ${1:label}, ${2:another} for placeholders.
 // Placeholders with the same ids are connected.
 // Example:
 // "Print to console": {
 //  "scope": "javascript,typescript",
 //  "prefix": "log",
 //  "body": [
 //   "console.log('$1');",
 //   "$2"
 //  ],
 //  "description": "Log output to console"
 // }
"logger_proc" : {
 "scope" : "plsql",
 "prefix" : "logger",
 "body":[
  "procedure ${1:proc_name}("
    ,"p_param1_todo in varchar2)"
 ,"as"
    ,"\tl_scope logger_logs.scope%type := gc_scope_prefix || '${1:proc_name}';"
    ,"\tl_params logger.tab_param;"
 ,""
 ,"begin"
    ,"\tlogger.append_param(l_params, 'p_param1_todo', p_param1_todo);"
    ,"\tlogger.log('START', l_scope, null, l_params);"
 ,""
    ,"  ..."
    ,"  -- All calls to logger should pass in the scope"
    ,"  ..."
 ,""
    ,"\tlogger.log('END', l_scope);"
 ,"exception"
    ,"when no_data_found then"
 ,"  logger.log_error('No data found', l_scope, null, l_params);"
 ,"  raise;"
 ,"end ${1:proc_name};"
  ]
  ,"description": "Logger procedure definition"
}
,"logger_log" : {
   "scope" : "plsql"
  ,"prefix" : "logger"
  ,"body":[ "logger.log('$1', l_scope);"]
  ,"description": "Logger single line"
}

,"logger_log_val" : {
  "scope" : "plsql"
 ,"prefix" : "logger"
 ,"body":[ "logger.log('$1'||$1, l_scope);"]
 ,"description": "Logger value"
}

,"logger_scope" : {
  "scope" : "plsql"
 ,"prefix" : "logger"
 ,"body":[ "gc_scope_prefix constant varchar2(31) := lower(\\$\\$plsql_unit) || '.';"]
 ,"description": "Logger scope prefix"
}

,"logger_start" : {
  "scope" : "plsql"
 ,"prefix" : "logger"
 ,"body":[ "logger.append_param(l_params, '${1:param_name}', $1);"
          ,"logger.append_param(l_params, '${2:param_name}', $2);"
          ,"logger.log('START', l_scope, null, l_params);"]
 ,"description": "Logger start lines"
}

,"logger_updated_rows" : {
  "scope" : "plsql"
 ,"prefix" : "logger"
 ,"body":[ "logger.log(sql%rowcount||' ${1:param_name}', l_scope);"]
 ,"description": "Logger updated rows"
}

,"logger_updated_end" : {
  "scope" : "plsql"
 ,"prefix" : "logger"
 ,"body":[ "logger.log('END: '||sql%rowcount||' updated', l_scope);"]
 ,"description": "Logger updated end"
}

,"logger_variables" : {
  "scope" : "plsql"
 ,"prefix" : "logger"
 ,"body":[   "\tl_scope logger_logs.scope%type := gc_scope_prefix || '${1:proc_name}';"
            ,"\tl_params logger.tab_param;"
]
 ,"description": "Logger variables"
}

,"apex_url" : {
  "scope" : "plsql"
 ,"prefix" : "apex"
 ,"body":[   "f?p=&APP_ID:&APP_PAGE_ID:&SESSION.:&REQUEST.:&DEBUG.::::"
            ,"f?p=App:Page:Session:Request:Debug:ClearCache:itemNames:itemValues:PrinterFriendly"
]
 ,"description": "APEX URL"
}

}

The trickiest bit was escaping the dollar signs for plsql_unit. Thanks Stack Overflow.

Perhaps this set of JSON could be transformed into the Intellisense format. That would make an interesting SQL exercise...

Next step, start organising my own stuff on GIT, instead of blog posts and other documents.

Monday, 23 April 2018

On Switching Code Editors

Ever since I joined the industry, my preferred text editor was TextPad.
It's a solid tool, similar to Notepad++. I probably underutilise it, but I started to see some people to interesting things with more modern editors.

Day-to-day, I work with two monitors, so I've been trialling software literally side-by-side for a while.

With some help from my nearby colleague, I tried a little Sublime (freemium), and that peaked my interest when contemplating the jump. Atom was also emerging at this time, which also had an impressive list of pros & cons. I wanted to give them all a fair go, but it can be tricky when you're busy, you know, programming.

I've ended up using Atom for a number of months now, even after moaning on twitter about it a few times. I wouldn't be surprised if it was over a year. I like the taste it gives for a modern editor, and it really intrigues me how it was built (JavaScript driven), but I think therein lies the downside. It's improved, but it doesn't scale well - for large files, or large lists of files.

Recently I've been hearing mostly good things about Visual Studio Code (free), especially for/by APEX developers, but also from the community in general. The young blood in the 2018 Stack Overflow survey regarded it highly - certainly far more than the 7.2% recorded in 2016.

2018 Stack Overflow Survey - Editors of choice

So I finally thought right, I've got to give VSC a good crack. There's stuff here I need to learn. Stuff that can make me more productive.

I printed a keyboard shortcut cheat sheets, since that's really what's going to make me productive. Luckily, like Toad & SQL Developer, Atom & VSC share a lot of shortcut commands.
And this thought in the back of my mind, perhaps from Editor wizards Adrian Png or Jorge Rimblas, that you should commit to the tool instead of using an extension to bridge shortcut commands.
and I wanted some new wins that would exhibit immediate ROI over the other editors.

I wanted to compile from my editor.

I realise that's what SQL Developer is kinda built for, but we also need dedicated editors for other document types.

I tried making this happen a few times with Atom, but I'm not sure it really liked our network structure.

I got this done in about five minutes with help from Morten.
No hiccups found, and I tweaked it slightly to use a "recent_errors" view I already had defined.

And with the extra time & excitement I had, I had a lot of fun with snippets - which this post was supposed to be about. I think I'm going to like snippets. Something I know Textpad had but I never got going with, but VSC is making the whole experience feel good.

I even find the syntax highlighting pleasing to the eye, so I think this is going to be a welcome transition to VS. Though I haven't played with CSS/JavaScript yet.

Using extension xyz.plsql-language

I understand it shared some framework concepts with Atom, so I'll probably get more familiarity with JSON while I'm at it.

"User Settings" is a place a developer can truly call home. While I yearn for a GUI, I see the advantage & flexibility with this approach. Though I'm sure it could be GUI-fied a little further.

Change default on left, added & overwritten by right.

I know a few actions I still do regularly in TextPad, so we'll see how things go on day two.
- searching across files
- basic macros for repeated text commands, though possible circumvented with the advanced multi-edit mode.
- compare files. I'm not sure why I still do this in TextPad.
- when I just want to see files in white background, with really familiar highlighting.

And with this momentum, I really need to restore my love for SQL*Plus with SQLCL.

If you want some more reading, I also came across a couple of articles that helped validate my experiences, and consider the move:
Switching from VSC to Sublime
Best Text Editor?

Scott

Monday, 12 March 2018

Emulating touch devices for Oracle APEX

In my Exploring AJAX presentation I had a frame that mentioned the ability for the Chrome browser to emulate the behaviour of a mobile device.

Open up the browser tools, I use F12. I spend a fair it of time in the Elements + Console tabs of these tools.
Then find the relevant icon, that will no doubt change location in future. Ctrl+Shift+M does the trick when the browser tools are open.

This mode allows you to change the type (size) of the device you're emulating; the orientation; and the network speed. Even the cursor changes to a shaded circle, to more closely simulate a (thicker) finger press.

Second icon from the left - follow the blue arrow

If you open your APEX login page when the emulator is already turned on, the user_agent will be spoofed accordingly. You could use Christian Rokitta's adaptation of categorizr to determine if you are on a 4" device, and populate an application item. That way, if certain regions don't fold well into small devices, you could conditionally replace them with more appropriate regions.

For example, I had a badge list of 5 values that didn't squeeze well into 4 inches, so I replaced it with a simpler value attribute pair region when my IS_MOBILE application item was set.
Note: don't forget to use declarative conditions where you can.

The 5.0 Universal Theme actually does a good job rendering regions responsively. Issues usually arise when there's simply too many columns to fit. I'm very much looking forward to the 5.1 OracleJET charts cathing up to to the touch behaviour. AnyChart prior to HTML5 doesn't adjust well.

Also available is the ability to throttle the connection. No doubt you have a pretty fine connection where you're developing, but the users on devices out in the field might not have the best connection. This setting allows you check your application's behaviour as if it were on a slower connection, highlighting any asynchronous issues, or lag that affects the user experience.

Nothing is ever the same as the real thing, so while it's a practical unit testing tool, you should still thrash out mission critical devices on the real McKoy.

I hear whispers that APEX 5.2 is going to potentially shake up the User Interfaces a little, with the removal of the jQuery Mobile User Interface. Perhaps there will be accommodations here to help our applications transition between screens, without needing dedicated pages?
I haven't seen any of the sessions delivered by the Oracle team talking up 5.2, so I'll have to wait and see.

Update -  I wrote this last paragraph a while back, though I don't think the early adopter covers any changes in this regard.

Monday, 29 January 2018

APEX Programmatic Spectrum

I listened to a session by Joel Kallman, and found myself thinking hard about this slide on the programmatic spectrum, diving into the concept of “low code” vs “high control”.

(From around 14 minute mark https://youtu.be/PZBXFjycoIo?t=14m26s)

Such a good representation of the spectrum of APEX components

I guess you might say tasks in the yellow end take longer to configure, and are difficult to maintain, particularly for someone new coming in, for instance:
At the green end, defining Authentication is very click-click-click, done. No code required.
Around the middle, using Authorisation schemes can also be declarative, but with a little tweaking, can give a decent level of control.
And to the right, for Custom authentication, while a typical package may look similar to the next, it’s driven with raw PL/SQL, but gives the developer high level of control.

Some other interesting claims in the seminar include APEX meeting the NIST needs of cloud computing from 2002. Remember, we didn’t even have Google Suggest until 2005.

And an application exported from 2008 still runs in 2017, many APEX versions later, looking and behaving the same.
As a developer familiar with the aims of the APEX upgrade process, this does not surprise me at all. What I didn’t consider was how much of a feat this is in the world of development.

Pretty impressive when considering version difficulties I've heard with products like JDev.
I realise that's comparing apples to toaster ovens, but it's a high bar that's set. Onya APEX team.

And the IDE is delivered via a browser :)

Friday, 12 January 2018

Data driven APEX icons

We have an application written with a heavily customised Theme 25 built for 10" tablets, and we feel the Universal Theme justifies the move, in part because of the surrounding ecosystem. Check out this forum discussion on the topic.

We've been looking through the packaged applications for applied ideas, and using the Universal Theme sample application as a component reference.

I came across a requirement where we had a list of items that indicated completion level out of 100.
Then I thought about a group of pie icons I saw in Dick Dral's Font APEX icon reference, and knew what I wanted to do.

All I needed to do was round a bunch of numbers to the nearest 5.

To solve such as problem, I usually start by giving myself a bunch of numbers to play with
select rownum rn from dual connect by level <= 100

Then defining that as a WITH statement, so I can refer to the derived column as often as I like.
with nbrs as (
  select rownum rn from dual connect by level <= 100
)
select rn
 , floor(rn/100 * 20) / 20*100 rnd
 ,'fa fa-pie-chart-'||floor(rn/100 * 20) / 20*100 icon
from nbrs
where mod(rn,5)=0
And I used a divide/multiply by 20 math trick to do the rounding.

We can then feed the result as the relevant icon into whatever region template we need.


I like pie.

Simple, yet effective.