Friday, 23 June 2017

Removing Outliers using stddev()

A colleague asked me about removing outliers from reports and I knew just the function to give a go.

Some time ago I had to prove that some data we had conformed to a normal distribution. Remember those from high school?
A normal distrubition (bell curve)
That request was about 2007 and I ended up using stddev() and lpad() to produce a vertical shaped bell curve in SQL*Plus. That was pretty cool.

As for removing outliers, we can use this tool to exclude a small percentage of whatever data you have, so that it doesn't skew averages.

If you want to look at the code in context, I've created a test case on

But as an inline reference, first I created a simple table, and populated it with a small set of random data between known range.
(I would like to add this is exactly what Tim Hall recently described here in regard to learning and asking questions.)

Importantly, I also inserted an outlier that was an order of magnitude higher than the rest.
My aim was to ignore this outlier in my SQL that calculated average.
create table outlier_demo  (a number, val number);

insert into outlier_demo  
select rownum -- we don't care  
  , round(dbms_random.value(20,50),1) -- some random numbers between 20 and 50  
from dual   
-- let's create 49 rows  
connect by level < 50

insert into outlier_demo values (0,500)

select * from outlier_demo;

A       VAL
---     ----
1       27.9
2       42.5
3       49.7
4       22.3
5       42.3
6       20.2
7       33.9
8       46.8
48      45.7
49      21.8
0       500
The average is higher than expected because of that outlier. I've included a column that calculates the average without the outlier, albiet hardcoded. Notice how close it is to the median? When considering usage statistics from APEX, I've often paid more attention to the median (most rows) than the average (affected by outliers).
select count(*) c   
   -- what's our average?   
  ,round(avg(val)) the_avg   
   -- what would the average be without the known outlier?   
  ,round(avg(case when val < 500 then val end)) no_outlier   
  ,round(stddev(val)) std   
   -- what's the median  
  ,round(median(val)) the_median  
from outlier_demo 

---  ------- ----------  -----   ----------
50        46         37     66           36
We can take the same function and get the result for every row, that's one way we can use it to filter outlier data.
select a, val  
   -- what is 2 standard deviations away  
   -- provide result for each row  
   ,round(2*stddev(val) over (order by null)) local_std  
from outlier_demo
Re-written as a from clause, we can now filter any rows that have a value higher than x standard deviations from the norm.
select count(*) c   
  ,round(avg(val)) the_avg   
  ,round(avg(case when val < 500 then val end)) no_outlier   
  ,round(stddev(val)) new_stddev  
  ,round(avg(local_std)) filter  
  (select a, val   
   -- what is 2 standard deviations away   
   -- provide result for each row   
   ,round(2*stddev(val) over (order by null)) local_std   
   from outlier_demo   
-- only average those values within two standard deviations   
where val < local_std;

--  ------- ----------  ----------  ------
49       37        37           9      132
This could be re-written in a form that may be more recognisable to some.
select count(*) c
  ,avg(val) the_avg
  ,avg(case when val < 500 then val end) no_outlier
  ,stddev(val) std
from outlier_demo
where val < (select 2*stddev(val) from outlier_demo);
There are probably a number of variations on how to apply this solution, and possibly a few alternative methods.
It will probably depend on the SQL in your particular report.

This is not to say you should be ignoring outliers completely, but stddev() can help identify them where necessary. And it scales.

Thank Kate for this post.

ps - If you want to dive deeper, Jonathan Gennick (my patient book editor) has two in depth posts on the topic, with some diagrams to help understand the mathematical concepts.

Wednesday, 21 June 2017

Annual Oracle release cycle coming?

Those of you on Twitter at the end of May, and certainly those at DOAG might have seen this announcement.

I'm no DBA, but that first sentence in the picture has the potential to be rather game changing.
Annual Feature Release of Oracle Database Software
More information is to come (from official channels), as so eloquently put in a comment on this only other post I've encountered on the topic, but I've been pondering this glimpse of change with some colleagues, here in the suburbs of our isolated Australian city.

We're navigating our way through a few upgrades now, and I've seen the same patterns before. Without a clear idea on event what month a major release may arrive, it can be difficult to engineer time into the schedule for an upgrade. It's like launching rockets, miss a window and you may have to wait months for another opportunity.

Other software vendors have set release schedules, and it an annual release cycle will no doubt influence the operations of Oracle consumers (and conferences) around the world, I think for the better.

And they probably won't need to worry about how to market the superstitiously tainted #13.

Wednesday, 14 June 2017

Presentation: Dynamic Actions, Javascript, & CSS for APEX Beginners

My first webinar didn't do so well, perhaps my new home will have suitable internet capability, in this "innovate nation" of ours...

My second one on my Boggex game went ok, though the room the locals were in was like an sauna.

My third attempt had it's own set of issues.
1) Ubuntu as an OS doesn't seem to enable broadcasting via GoTo Meeting.
2) The wireless policy in the Oracle building is effective, but not great for hastily setting up a second laptop
3) The navigation keys on Lino's keyboard was just different enough to be disruptive
4) While I use on Chrome's save password feature, I'm pretty confident with most of my APEX passwords. It's the username I was stuffing up on the foreign PC.
5) The think tank at HP thought it would be a good idea to map Aeroplane Mode to F12, so when I proceeded to display the Chrome browser console...

So balance was restored after the first 10 or 15 minutes, which rattled my first attempt at presenting using just an APEX app - no powerpoint/prezi.

Though I think the concept was fine. It's just like training people, jumping between Page Designer and runtime.

I'd give it another go, and it's less presentation prep. I learned a bit from the first session, and explored some layout options while I was at it.

I still baulk a little at the difference between talking with a live crowd, and a community of ears at the other end of my microphone. I wonder if (finally) creating a few more videos will fix that? Once my ultrabook is back from computer hospital, perhaps. In the new house?

As for the topic, I explored dynamic actions, JavaScript and CSS for those trying to understand the fundamentals. It's a favourite topic of mine and I think there are a few basic patterns that get me a long way in creating interactive applications.

No doubt I'll refine these example further, and how to communicate them more concisely. Stay tuned.

I've made the app available here. Please feel free to provide any feedback, either here or through the app.
My favourite page is P50 specifically on Dynamic Actions, probably the most versatile utility within APEX.
If you scroll down you'll see the relevant code I used to sustain an interactive page that doesn't need to submit.
Also look out for things like conditional buttons, conditional colours (foreground and background), APEX behaviour considerations, among others.

If you're in Australia and you'd like training on this topic, I could demonstrate cool stuff for hours ;p
I also aim to get some organisation around github sorted when I get my laptop back, but I've at least put the export here so you can check it out directly.

I hope you learn something new.


I'm still here.

It has been some time since my last confession.
It's been a while, but sometimes you get busy, ya know?

And a different sort of busy, the sort that left me wanting to escape to a different place. I spent a bit of time playing an old favourite, Civilisation.

I really like to write, consolidate ideas, and share insights in this forum, but it's rare that I have such a gap.

I also miss it.

I always have plenty of unfinished drafts, but I also usually have a few recent experiences that are like an itch waiting to come out.

There are also a few other factors. My laptop is just within warranty and I'm finally getting a line of pink pixels fixed. I'm also gonna wipe Ubuntu off it and go back to Windows.

I feel my OS sojourn was successful in that I realised in a career of learning such as development, some of us need to learn to swim in a pool, not the ocean. Not the best analogy, but close enough.

My workplace is always (usually) Windows, any exploration on my laptop can be within a VM from the comfort of Windows. I've also got accumulated decades worth of keyboard shortcuts, and the muscle memory alone keeps me to the original style keyboard, not the weird ones that Macs use.

And it turns out we're also gearing up to move house, well, start by putting our place on the market. That sounds like fun, right?

You'll find me escaping occasionally, back here, writing with passion about what I find to be an interesting career. That, and reading/listening to stuff about science. There is some amazing stuff going on in this world!

Wednesday, 29 March 2017

Frequent APEX questions

I've put a call out on the OTN APEX forum asking for suggestions from the community on what they think are frequent APEX questions, be it within the forum or elsewhere.

In part, I'm chasing ideas for content in future presentations, or blog posts. I welcome others to do the same. I think if we share our perception of common questions, better resources will come.

So please, feel free to contribute in any way, shape, or form.

What questions do you see or hear all the time?

If for some bizarre reason you don't have an Oracle account for the OTN forum, just reply to this post, ... then register an account. It covers you for the forums,, the Dev Gym, entitles you to download stuff like SQL Developer, and no doubt more.

Though probably no steak knives.

Make Radio Group look like UT Buttons

It's pretty easy to convert links in APEX reports to use the Universal Theme button look & feel, as I described here

We can apply the same technique to radio groups, but it requires a little more work than just adding some classes, but nothing like jQuery mobile radio fiddles.

It took me a few goes to get the correct string in the correct APEX attribute to make the individual buttons look & behave the way I imagined, so I thought I'd share the journey. Show my working, so to speak.

First, create a radio item:

A humble radio group

Then set 'Option HTML attributes' to add template option classes, just like I did for the report action buttons. This applies the string to each option, rather than the entire item.

class="t-Button t-Button--simple"

If we check the page, the radio group should have button boxes surrounding the text labels.

Radio group with button class

To work out what CSS selector I needed to hide those radio buttons, I used Chrome's Inspect element tool (right click a radio button) so I can see how the HTML is constructed.

Inspect Element is the best browser tool. The best.

One shared property for each item is how the input tag has type=radio.
Further up you can see the entire fieldset as an id of P47_RADIO, our item.
Combine these to identify any radio buttons within this specific item.

Edit page 'Inline CSS' to using this combined CSS selector, with a property to hide any web component identified with that selector.

#P47_RADIO input[type=radio] {display:none;}

This tidies the button group, trouble is,  the current selection is unknown without the native radio button. This can be solved by creating a dynamic action that toggles button classes upon click of each radio item.

Create dynamic action on click of jQuery selector

#P47_RADIO span.t-Button

Add action to Execute JavaScript
// add simple class to all options
$('#P47_RADIO span.t-Button').addClass('t-Button--simple');
// remove from current selection
The result:

In my case the t-Button--simple template option is removed from the current selection, but you can play with these options to choose a combination that suits your desired contrast, using the various colours:


Or "hotness", a highlight:

Or contrast, well, not highlighting the entire button:
All demonstrated in the button builder reference in the Universal Theme app.

You can achieve consistent button width by adding this to your Inline CSS:
#P47_RADIO t-Button {width: 100px;}
After your page loads, you can use the inspect element tool and increase the width 'live', to find a figure/unit that works for you.

Now we need to ensure this selection is defaulted when the page opens. I prefer to execute the click() event as to simulate the actual action, as opposed to replicating the relevant addClass() function.

You can use a similar technique to conditionally hide a specific radio button in the group (though you still need to validate the availability to the user on the database end.)

Add this to your 'Execute when page loads' page attribute, instead of adding a Default Value the item itself.

$('#P47_RADIO input[value=P]').next().click()

Where the value=P represents the relevant radio item value. The .next() moves selection to the next sibling, which is the span element, where click is triggered.

Or you could consider this button group plugin.

Or in APEX 5.1 you can use the pill template option.

Or you could use some form of List template button group.

Or you can stick with tiny buttons only people using a mouse on a desktop PC can hope to aim for.

One of the beauties of Oracle APEX - so many options.

Happy APEXing!

Monday, 13 March 2017

Font APEX between versions

Sometimes, it's the little things in an application that make users happy.

Sometimes, it's just the icon on a darn button, card, or menu that makes all the difference.

Surely by now you've encountered using icons within Oracle APEX, and we've come a long way since Theme 25. To get anyone up to speed, APEX 5.0 saw the introduction of Font Awesome baked into the builder, which is a reputable CSS icon library for such things.

5.1 vs 5.0
As with other 'plugins', Font Awesome library releases new versions with new icons at a different schedule to APEX releases. It's easy to get your 5.0 instance up to speed with the latest library, thanks to Patrick's simple write-up.

Now the same goes for the 1000+ icons that come with Font APEX, some of them database specific which for some reason excites me a little. If you need to catch up, Dick Dral has a great low down on what makes Font APEX awesome, so to speak, and for those who are familiar with Dick's work, that's a double down on word play - see his URL of his sample application, which is growing to quite a utility.

But if you would like to use the latest library, or even use it within APEX 5.0, Max Tremblay gave us this great post. It's a little more involved than Patrick's, but things aren't as easy.

The long snapshot of menu icons shows the difference between 5.1 on the left, and 5.0 on the right. They've cleaner, they scale better, and we can no doubt all thank Shakeeb.

Max provided some CSS that's required to suit adjust usage to Font APEX, but having an app with icons found all over the place, I encountered some other components that needed adjustment. With some further back & forth on the APEX slack channel, Max helped me come up with these.
/* Font APEX 
CSS required to make Font APEX from 5.1 work in 5.0
Headstart provided by Max Tremblay

/* Most icons around the place */
.t-Icon[class*=' fa-'],.t-Icon[class^=fa-]{
    font-family: font-apex!important;
    font-size: initial;

/* Left side menu */
.t-TreeNav .a-TreeView-node--topLevel>.a-TreeView-content .fa{
    font-size: initial;

/* Left side sub menu */
#t_TreeNav ul ul .a-TreeView-content .fa {
    font-size: 12px;
    padding: 10px 0;

/* Navbar dropdown */
.t-NavigationBar-menu .a-Menu-content .a-Menu-item .fa {
    font-size: 12px;

/* Navbar top row */
.t-NavigationBar-item span.t-Icon {
    padding-right: 5px;
    padding-top: 0px;
    font-size: 14px;

/* Slide tooltip plugin */
.a-DetailedContentList-icon {
    padding-top: 10px;

/* Custom icon usage within breadcrumb */
.t-Breadcrumb-label .fa {
  padding-top: 3px; 
I think this once again shows the versatility APEX has between versions, quite literally replacing one plugin with another, with a little help from the community.


Saturday, 25 February 2017

Understanding the User's Perspective

Recently I encountered what I thought was an interesting allegory in regard to software development and understanding the user experience.

My three year old approached me with her bottle of milk and requested more, insisting it was empty.

Exhibit A - One non-empty receptacle

I assessed what was presented to me:

  1. There was milk in the container
  2. The straw reached the current milk level
  3. The straw was not blocked (I didn't feel like milk. I did not enjoy this test)

It worked for me.

So I returned the bottle and went back to ... cooking dinner. Right, let's go with that.

The user child came back telling me the page bottle still wasn't working, so I asked for a demonstration.

Of course, the user was doing it wrong.

All programmers know this is the eventual outcome because the user is always clicking the wrong buttons at the wrong times. The real challenge is working out what to do next.

In this case, I filled up the bottle with more milk. Any other outcome with a boundary-pushing young child will not be acceptable.

However, once gratified, I also educated the user with not only the appropriate use but why that's the case. I think the reason "why" is as opposed to "just because" is partially what's wrong with society's ability to comprehend and assess new information.

Have you worked it out yet? 

She was turning it upside-down to have a drink. This meant thanks to gravity, the milk was no longer near the end of the straw, and it was deemed 'empty'.

I figured she turned it upside-down because that's how most of the other drink bottles work. Then I realised that's not accurate. Most of her bottles involve sucking liquid up a straw. I'm not sure why she often makes this mistake with this particular bottle, but I figure if I teach her some basic physics, not only can she potentially work out the problem herself next time, but maybe even apply the reasoning to other problems.

So perhaps two lessons

1) Watch your users. Don't be creepy about it, but somehow I'm always stunned by a number of notes I make when watching users operate software.

2) Educate your users. There are so many ways you can give your users cues as to how to use your software, but the best-designed tools don't need instructions. They're inherently intuitive.

When testing our own software, the more we can think like users instead of developers, the better our product will become.

Sounds like an old adage, but what do you think of the analogy? Have you encountered similar ways to relate to non-IT folk?

Tuesday, 7 February 2017

2016 Blog Review

It's February, so it must be time to do this. I explore what thoughts arise from looking back, and forward to the future. It helps me to remember stuff, decide what to do, and apparently some people find it interesting.


So many tools. And I arguably don't use enough of them.

APEX 5.1

It's out and I'm excited. Unfortunately, scheduling is super tight, so I'll be waiting at my current site until hopefully around June, but I've certainly been enjoying the play on I look forward in particular to the OracleJET built-in charts, we've got some great dashboarding ideas to explore.

I've seen plenty of questions come through the forums on the Interactive Grid, but I'm by-passing most of those until I get the chance to use it. I sure know where it belongs, and I look forward to understanding how it ticks, and the implications it brings.

JET plug-in - work in progress

OracleJET Visualisations

After an interesting overview in the depths of OracleJET from Chris Muir, I assigned myself homework that's seen me quietly working away on some plugins, bringing OracleJET visualisation (charts) for use in 5.0. I'm not sure how successful the cultivation will become, but I am documenting my journey. Expect a decent series of articles on my process, but be patient. I've had a hiccup with JSON and I'm giving it a rest for a little bit.


We have 12c at our main client site and I appreciate some of the features it brings, so much so I look forward to 12.2, particularly LISTAGG.

I particularly like Identity columns; the row limiting clause, though I probably abuse it sometimes; and I reckon I must spike page hits for Tim's post about lateral & outer apply joins.

I'm starting to explore performance benefits of PL/SQL in WITH clause, and the UDF pragma, among other features.


My colleague has done some interesting stuff with parsing incoming JSON using 12.1 SQL, and I'm slowly exploring with my plugin generation using APEX_JSON and LISTAGG. With made another step deeper into 12c by upping MAX_STRING_SIZE to allow for processing of larger JSON.

I'm quite glad it's no longer XML.

It seems 12.2 will be bringing the remainder of tools required to really get going with this. There's quite a lot of polish I've seen in this release that makes it an attractive upgrade.


After many, many years using Textpad, I'm giving other editors a go. Atom has shown good promise, but I'm still yet to get it compiling my packages on a Windows box, though I haven't persisted.
There are some other minor niggles, but it does bring great benefits. I might give Sublime a fair go this year, however. There's beer in it for anyone who can get me compiling from a text editor that pleases me. Sorry, I only use SQL Developer for ad-hoc queries, compiling from scripts, and some not enough built-in reports.

I am casually interested in how Atom was built. I see Android Instant apps as being something to keep an eye on with this amazing JavaScript tech.

JavaScript and node.js

I really haven't given node.js a fair go, and if there was anything other than OracleJET visualisations that I want to learn this year, it's node.js. Maybe my QNAP will help me learn after all.


As much as I see the benefits of running a linux based operating system, I think I just need to accept my fate as Windows proficient, using Virtualbox when I can. Too many hurdles to transition operating systems when there is a world of user interface development to keep up with.


I can't add much to this from this year, except we've had a steady keel for 12 months. I'd like to move to nearly whatever the current version is when we up to 5.1.
I did notice spikes in the connection pool that endangered other users when a page was opened containing an image gallery, particularly for a 'job' with a large number of associated images. We added pagination to that page to cap these spikes and keep them below our connection pool thresholds.


Thanks to all those who answer and ask questions online (and at conferences, of course).


Plenty of hub-hub this year about the Oracle ACE program, and the potential for a number of Ace Alumni to appear. It's going through a maturation phase. Let's wait for it to evolve and continue to recognise and potentially aid those people helping the community thrive and develop. Kudos to all those diligent experts out there who remain unrecognised within this particular program, but support their teams and community with their humble expertise.

I'll continue to output stuff that helps me remember stuff and benefits other people, regardless of which side of the line I'll fall. Whatever helps my abstracts get submitted when I occasionally venture out in this giant planet ;p


No real exciting stats to report this year. Growth minimal, but quite the regular heartbeat of visitors. Many of them me, looking up certain references. This is the very reason that tipped me over the edge when starting this blog, seeing this byline:
"Oracle Things I Got to Remember Not to Forget" - Alex
2016 Blog visitors - Lift your game, Greenland

For some reason, my reference to Carsten's LISTAGG function was the most popular this year. Perhaps because more people like us who are using this to generate larger JSON sets prior to 12.2?

Also up there for hits was about one of my favourite APEX development patterns.

It took until #22 in the most visits by page for a 2016 post, an important one on improving PL/SQL performance in APEX. Some of my favourites from this year include decommissioning triggers, and a debugging how-to that may be a useful reference in the forums (#40).

Spiking briefly was this Patterson-Gimlin style glance at APEX 5.2

APEX Sample Applications

If you haven't heard, Dick Dral possibly leads the pack at the moment in regard to sample applications, I need to have more of a play in there. He's been a busy boy!

While reviewing blog page hits I noticed activity this page where I started to catalogue my bookmarked list of community sample applications. Maybe I can finish modernising my sample app and make these more prominent, or find a better home elsewhere.


For anyone who purchased a copy of my jQuery in APEX book, thank you, and I would recommend you re-source chapter 9 electronically. Somehow an early draft made it in but it has since been replaced, using what seems like a logistically amazing process in the world of publishing.

If you like video format, I did this video series for 4.x, but many principles still apply.

While I've got a back-log to read, this Real World SQL and PL/SQL is hard to go past. Well done, gang.


Because I like science and technology.

Among other things, I predict forms of intelligent systems / AI will be a science we'll see more of in the world, solving problems that wouldn't immediately spring to mind. I also wonder if it will help clients become event thinner, so I can stop complaining about not having enough space for apps on my phone, even as a moderate user of apps.


Because I've liked this since I was single digits.

This year will mark the end of an amazing mission that I've been following for quite some time. Cassini has been exploring more than just Saturn as shown in this amazing timeline. It's already committed on its fateful descent into Saturn in September.

Saturn moon Iapetus

I know in next year's review I'll be talking about the amazing James Webb Space Telescope, scheduled for launch around a year after Cassini farewells. This thing is going to be the biggest thing for astronomy since Hubble. Literally. The few weeks between launch and confirmation of a functioning telescope will be the most thrilling moment that represents years of work for thousands of engineers. There's no second chance like Hubble had, nor any service missions, not while still humanity struggles to exceed low earth orbit.

Science, in general, is actually quite a big part of the current news cycles, but for all the wrong reasons. I wonder how this will change if China's Chang'e 5 makes it to the moon and back this year, making it the third country to achieve this amazing feat and the first in 40+ years. I wonder how POTUS will respond?

There is at least one reason to be in the land of the free this year, for on August 21 a total solar eclipse travels across the entire country, a rare event hopefully inspiring many budding young scientists.

Happy science, everyone.

Thursday, 2 February 2017

How did you get into programming in the first place?

If you want a sanity check on programming life, I recommend you follow @ThePracticalDev in some form.

Recently a question was posed regarding we all got into programming in the first place.
There are a few interesting posts, here is my submission.

The Vic-20 was the first introduction, using BASIC pokes recorded on tape, but I think it was seeing the use of a variable in a simple Pascal programme over the shoulder of another student in year 9 that sparked something for my career choice. Ada at university, then PL/SQL with Oracle databases.

I often forget those first years, in between playing some form of space invaders, where I could follow a spiral bound manual to write simple BASIC programs, recording them onto a magnetic tape recorder.

Vic-20 "Datasette"

Insert many years of playing with DOS and autoexec.bat; playing Dune, Civilisation, and any number of cold war games & flight simulators, I found myself in a computing class when I was early teens. In maybe our second workshop I noticed the nerd in front of me doing something interesting with a 'variable'. No doubt my memory has warped this moment over time, but I vividly recall contemplating the creative implications of this temporary memory storage.

This nerd & I became friends, we created a cool little character-based word-sleuth game using Pascal, and I think I was kidding myself thinking of a career in Architecture.

It helped that Ada was the language of choice at the time in university. Look familiar? We finished our project work using Oracle, and I found myself at an Oracle client site instead of building internal systems for Collins class submarines...

Now I blog stuff about a revolutionary front end IDE built on top of amazing database technology.

What will we all be programming in three more decades time?

Monday, 30 January 2017

APEX Survey Results: Addressing Performance

Yep, I'm still doing this. A bunch of questions to come, many worth the visit. Just a few weeks between drinks, so to speak. My annual review is a little late, too. Anyway...

Time for the performance questions in my 2015 survey. A favourite topic of mine, and my boss, Penny Cookson, lives for tuning.

Q9. How do you proactively address performance? (tick all that apply)

Tune SQL - I would be surprised if this wasn't the top result. Plenty of SQL used in APEX applications, and why not tune them?

Limit Interpreted PL/SQL - this is good practice in general, but in the world of APEX, this means moving inline code from application into packages. You can get pretty quick wins doing this with any plugins. Read here for details.

Materialised Views - a database construct that can aggregate complex information at regular intervals, to be queried many times with simpler SQL over fewer rows. I've seen interesting examples that obfuscate layers to external information using pipelined functions. Or you could just use it for your menu.

Care using v() - Particularly important when referencing page items, since this function would execute a query on the session state table. While values such as APP_ID come from a persistent package variable, it's still context switching between SQL and PL/SQL, so it would be even better to use bind variable.
where id = (select v('APP_ID') from dual)
I think any use of v() should be questioned. SQL queries should use bind variables (not substitution strings) and packages should be parameterised.

Page Workflow/Design - why refresh the entire page when a partial refresh would do? A well designed application will limit network traffic and the amount of queries necessary to serve the data. Refreshing regions on demand with dynamic actions is one of the most regular things I do. Declarative page modals in APEX 5.x have also made APEX life a lot more comfortable.

Global Page Modularisation - Performance also about the developer not repeating tasks over and over, causing future maintenance headaches.

JSS/CSS File Management - minifying code can reduce network traffic, as can well designed libraries; CDNs; and declarative options. I understand APEX Front End Boost can help.

Care with jQuery selectors - selectors can be abused just like table indexes. I have some commentary on this here.

Region Caching - possibly underutilised, but the ability to cache region on the global page in APEX 5.0 opens options, as do some new APIs.

data-attributes - jQuery related, what I meant by this was to offer more information during report generation, which can help interactivity and reduce AJAX activity. Though read consistency issues should be considered. See a basic example here.

Other - a few people suggested a well designed data model. Touche. Even more got stuck into it in the next question on performance return on investment.

The User Interface attributes are also an area worth being familiar with, particularly in regard to application level file management.

I don't need to worry about performance - yeah, right.

Friday, 27 January 2017

Truncating LISTAGG prior to 12.2

I've been following the LISTAGG problem for a while because I find the function useful.

The problem being this error:
ORA-01489: result of string concatenation is too long

It relates to when the result set is over 4000 characters, eg:
with data as (select rpad('x',400, 'x') str from dual connect by level <=10)
select listagg(str,', ') within group (order by null)
from data
Ten rows of 400 characters equals 4000, add the concatenation operator and you get the error.
If the second parameter was null, this would actually be within the 4000 limit.

There are improvements in 12.2 to handle this overflow.
with data as (select rpad('x',400, 'x') str from dual connect by level <=11)
select listagg(str, ',' on overflow truncate '...' with count) within group (order by null)
from data

The output would end

But if you're still in 12.1 or earlier, you need other options

One solution is a custom LISTAGG function to return a CLOB.

You could make a decision on the incoming length, which probably isn't always practical, or you can concatenate only the first x rows using a CASE statement
with data as (select rpad('x',400, 'x') str from dual connect by level <=10)
select listagg(case when rownum < 5 then str end
              ,', ') within group (order by null)
            ||'First 5 of '||count(*)||' shown.' as str
from data;

Which ends the output with
...********First 5 of 10 shown.

I suspect this will do for a lot of scenarios. Ordering is up to you.

I don't remember encountering this idea in the context of LISTAGG, but it seems nice and simple.

LiveSQL example:

Tuesday, 24 January 2017

Escape Special Characters APEX Demo

A few weeks ago I wrote more detail than expected regarding escaping of special characters.

I thought I'd add a simple demonstration, for reference.

Consider the following query, with variations of escaped column output.
with data as 
  (select q'[G'day,]'||chr(10)
           ||'<br>APEX<script></script>' as string 
   from dual)
 -- UI default
 string dflt
  -- where no tags expected
 ,apex_escape.html(string) protected
  -- good for most things
 ,apex_escape.html_whitelist(string) whitelisted 
  -- replace line feeds with HTML line break. Could use chr(13)
 ,replace(apex_escape.html_whitelist(string),chr(10),'<br>') protected_custom
 -- UI naughty
 ,string naughty
from data
The major difference being unescaped output looks like
Scott <strong>loves</strong><br>APEX<script></script>"

While escaped output looks like
Scott &lt;strong&gt;loves&lt;&#x2F;strong&gt;&lt;br&gt;APEX&lt;script&gt;&lt;&#x2F;script&gt;"

It determines how the browser will interpret these tags and display them to the end user.

Note that the first field called "DFLT" is using the default setting, therefore escaping special characters.
All other four columns do no explicitly escape these characters, deferring protection to the SQL.

Colum Attributes across four columns

The output looks like the following. The first line may look similar to any time you attempt to use APEX_ITEM in a classic report without turning this flag off.

Sample output, using template: Value Attribute Pairs - Column

So depending on what you're trying to display, you might need a particular combination of code / settings.
  1. Default - default APEX behaviour, no column settings adjusted.
  2. Protected - uses apex_escape package to do the same job as declarative attribute
  3. Whitelisted - Certain markup tags are allowed, but all others are still escaped
  4. Protected custom - often I want to replace line feeds/carriage returns in data with HTML line breaks. This combination facilitates the best of both worlds
  5. Naughty - avoid unticking Escape Special Characters attribute without protecting data within SQL. This is enabled Cross Site Scripting (XSS)

The naughtiness can be demonstrated by adding alert("Hello universe") between the <script> tags. The unescaped column will mean the browser will render an alert when the page renders.

This is bad because instead of an alert, it could be some malicious JavaScript.

Further security tips can be found at Recx.
APEX-SERT is also 5.0 ready.

Thursday, 19 January 2017

Re-evaluating APEX Authorization Schemes

Authorization schemes in Oracle APEX are used to control access to page, buttons, and all sorts of other components.

In my experience, these are best defined at a privilege level, where the same privilege could be allocated to multiple business roles, but that's for another post.

In this post, I want to mention a cool API function called apex_authorization.reset_cache, which helps control the behaviour of these authorization schemes.


While googling something else I stumbled across an interesting function called apex_util.reset_authorizations, only to find it was deprecated in 5.1, replaced with the same (but renamed) function in another package.

APEX_UTIL was getting big, and even though the team was trying to manage it, 5.1 was the first time I noticed it reduce the number of procedures. Too many procedures mean some gold nuggets get lost, perhaps until they're moved to a more specific package.

I'd say we're all guilty of putting a procedure in the inappropriate package or letting a "utils" package grow too big. Me probably more than many, but there's been a lot of clean-up in 5.1. If there's any documentation you read about 5.1, let it be the release notes.

I also found the function mentioned in an old email I marked for blogging about because of an OTN forum post I was listening to. Not because of the original topic, but another deprecated/renamed procedure mentioned within.

Authorization Evaluation Point

The default evaluation point for authorization schemes is once per session. This means the first time APEX comes across a component protected by an authorization scheme, it will evaluate it and remember the result.

Authorization Scheme Evaluation Point

This default option is best for performance, but if you want to afford your users the ability to pick up changes in authorization without needing to log out then back in, then you can use one of the other options.

Re-setting authorization schemes at runtime

A cool API exists called apex_authorization.reset_cache. The documentation states
"it resets the authorization caches for the session and forces a re-evaluation when an authorization is checked next".

This means you can provide a button to the user that would clear the slate for all authorization scheme outcomes in that session and force APEX to re-evaluate any authorisation schemes it encounters.

This will be handy for me because it offers a chance to do this on demand at runtime, instead of once or 'all the time'. It also does the job across multiple applications that share authentication.

This is one of quite a few library functions now available in 5.1, in fact, this particular one has been around in some form since 4.2, but buried in apex_util.reset_authorizations.

Under the hood

If you want to see the interactions in the underlying APEX table, this query will help.
select s.*, a.application_id, a.authorization_scheme_name
from apex_050000.wwv_flow_session_authz$ s
join apex_application_authorization a
  on a.authorization_scheme_id = s.authorization_id
where session_id = 16678957299354
See, it's not magic. Just clever.


Providing Security through Authorization
APEX 5.1 Release Notes

Wednesday, 4 January 2017

APEX attributes for Escaping Special Characters

A relatively common on the forums is regarding the escaping of special characters in reports, but it seems the developer isn't always sure what is actually happening and how to how to search for it.

It seems I've had this on my "to blog" list since April 2015, but now that 5.1 has been released, it seems more people are coming out to leave 4.x can't work out where the Standard Report Column option is.

APEX 4.x Display As attribute

This was required when HTML was present in the query, either to add tabular items manually using apex_item, or to style data (though you should use HTML Expression instead)

Example of special characters being escaped

For instance, if you've written a query like so

SELECT APEX_ITEM.CHECKBOX2(1, empno, 'CHECKED') chk, ename
FROM   emp

And are only seeing the HTML code in your column output

<input type="checkbox" name="f01" value="7369" CHECKED />

Then you need to Escape Special Characters, now found in the Security section of the column properties as a Yes/No option.

APEX 5.0 Escape Special Characters attribute

This is defaulted to Yes to help protect from cross-site scripting (XSS), a common security concern in the web world where data entered by users is stored in the database, then when rendered it can be interpreted as HTML code.

Set to No to allow your data to be rendered as you may expect. 
Note that in the 5.0 component view this is still referenced as Display As - Standard Report Column.

The change in terminology is documented in the 5.0 release notes

Report column property naming differences
Please note that if setting this attribute to no, you should still make efforts to protect your applications by escaping data where possible. For example, if I wanted to replace all carriage returns with the HTML line break, you can still escape your data then add HTML content.

replace(apex_escape.html(card_title), chr(10), '<br>')

You could probably do a variation of this using apex_escape.html_whitelist

If you're combining two fields, separated by the line break:


then you might as well use HTML Expression and keep your data/UI layers separate.

HTML Expression attribute

Check out the open source project APEX-SERT to help find potential security concerns with your Oracle APEX applications.

See escaping examples in APEX reports here