APEX Web Service Request Limit

Every now and then there's a new setting worth mentioning.

The New Features page included news regarding the logging of

Requests to external web services from inside the database

Firstly, this only appears to log requests coming from within the APEX environment.
Requests from scheduled jobs will not be logged, as of 18.2.

Secondly, moving to the cloud can be ... a learning experience. Since we're now using Social Sign-in, web requests are up. We also re-configured directly listings and file fetches to use web services. Web services are great tools.

However, there's a new setting that needs guesstimating before you expect too many users to log in, otherwise you might spot the following error in your error logs:
ORA-20001: You have exceeded the maximum number of web service requests per workspace

Search for this message and you'll find Dimitri encountered it during his experiences with AOP. Pierre points also points out the setting in a forum post.

Found in the Internal workspace, under Security settings, Workspace Isolation, there is a Maximum Web Service Requests attribute. There is a workspace level setting under Manage Workspaces, Existing Workspaces, then choose your workspace.

Maximum Web Service Requests attribute

However, the instance level setting has a default of 1000. You may need to review that value if you have a reasonable number of users using web services within Oracle APEX.

Now while the item help states:

Enter the maximum number of web service requests that Application Express supports for each workspace in this instance. You can configure a more specific value at the workspace-level.

This forum response from Carsten states this applies to a rolling 24 hour window.

I immediately envisaged a JET chart in our monitoring application, so we could visualise how this figure changes over time - and to see where it peaks. After a slight hiccup, I now see this graph that tells me out limit should at least 4000, to allow wiggle room.

Plotting apex_webservice_log

The really thin bars were an accident that I ended up thinking better represents the data, as compared to being a line graph.

So as activity occurred during the day, as depicted by the green line, the rolling 24h figure in blue crept up. This was calculated using an analytic function that allows me to create a moving window on the data based on an interval of one day. And I'm summing how many at each moment, hence the sum(count(*)).

select * from (
    select request_date 
       ,sum(count(*)) over 
            (order by request_date  
             range between NUMTODSINTERVAL(1,'day') preceding 
                       and current row)  last_24hr
    from APEX_WEBSERVICE_LOG 
    where request_date > sysdate-3
    group by (request_date) 
-- must ignore first day in inline query
-- as cumulative range will be skewed
) where request_date > sysdate-2 
order by 2

The 'Hits' count in the y2 axis is rounded to 30 minutes, finding inspiration from these forum responses.
trunc(request_date)+round(to_char(request_date,'sssss')/1800)/48
Without this rounding, the chart looked too much like a NY skyline, while the rounding smooths the staccatic nature of the data. Perhaps there's a JET attribute I could use instead?

What took me some convincing was the shape of the rolling average graph at night. I knew daily activity was essentially in the shape of a bell. I saw it in the y2 axis above, and I see it in the APEX activity logs.

Two days of APEX activity. Can you see lunchtime?

So I found myself building a test case with a few dozen rows representing the bell, just to check there wasn't necessarily a dip in the rolling data overnight, instead of flat-lining.

I'm was not entirely convinced, so I detailed a the test case in this post, looking closely at the range analytic function. Get some real "SQL-perts"? on the case. It's the same query as above, just with smaller, static data set.

And now I've seen the rolling activity drift down on Saturday mornings, I'm more confident with the graph outcome. It's almost poetry.

What happens on the weekend

I must say, playing with these time-based graphs has been a much easier experience in 18.x. I'm still getting the hang of which settings to use (particularly with dates), but at least I can just use the SQL I need, not extra bloat just to help the UI.
While I'm complimenting these charts, the database is also thankful we can now define the SQL once, and use the same source to plot multiple series.

Load test your applications, people.

Visualising SQL Analytics Rolling Count with OracleJET in APEX

Back in around 2005, before the time of smart phones, I had some data.

I can't remember what the data was, but I was told that for it to be valid, it should roughly form a bell curve.

Sure, I'm sure I could have aggregated it, exported it to Excel, and plotted it to a graph, but this was 2005. There was no SQL Developer where I could copy & paste the results directly into Excel. No panel in Excel with a button that plots that data in one click. Back in my day, thing like that were a little bit more work ;p

This was a time when I enjoyed SQL*Plus, and I ended up writing a query to nest STDDEV within RPAD so it would plot asterisks down the page, in a vertical bell. In was a work of art. I'm sure I've still got the test case labelled in bowels of Gmail.

Fast forward to today, where I have data I know starts as a daily bell curve, but I'm applying a rolling 24 hour total over that daily resonance. How does that affect the wave?

I use today's fancy tools to plot the said data, and it doesn't turn out quite how I pictured.

This is my query below plotting real data using OracleJET charts in Oracle APEX.

The underlying data still formed a bell, more detail in a future post, but smoothing this out to a rolling 24 hour total - what should that look like? Why is there a flat line during no activity, instead of a dip?
No doubt there is a math-geek corner of the internet that explores this pattern, but I still wasn't confident with the query.

Real data is awesome. The trouble is, it can be real noisy. If you're trying to reconcile your results, it's often worth building a simple test case. As a regular forum participant, I'm such an advocate for this. If the person asking the question took the time to write the DDL for the test case, I reckon half the questions wouldn't need asking. The other half will have an executable test case ready for any responders.
How's this - in a response to a question I asked (with a test case, of course), Andrew Sayer responded with a LiveSQL demonstration! AskTom questions that have a livesql.oracle.com example are generally prioritised.

So, what I have here is a real simple table, with some "logs" at specific times, and a little variance. It's quite similar to the setup you might see for SQL exercises in the Oracle Dev Gym.

-- create a simple table of dates
create table range_eg  (rd date);
-- ready for adjustment
truncate table range_eg;
-- insert sample hits at a regular interval, with more frequency around noon.
insert into range_eg values (timestamp '2019-03-26 09:30:00');
insert into range_eg values (timestamp '2019-03-26 10:30:00');
insert into range_eg values (timestamp '2019-03-26 10:45:00');
insert into range_eg values (timestamp '2019-03-26 11:30:00');
insert into range_eg values (timestamp '2019-03-26 11:45:00');
insert into range_eg values (timestamp '2019-03-26 12:30:00');
insert into range_eg values (timestamp '2019-03-26 12:45:00');
insert into range_eg values (timestamp '2019-03-26 13:30:00');
insert into range_eg values (timestamp '2019-03-26 13:45:00');
insert into range_eg values (timestamp '2019-03-26 14:30:00');
insert into range_eg values (timestamp '2019-03-26 15:30:00');
insert into range_eg values (timestamp '2019-03-26 16:30:00');

-- I took a few hits out the middle, so the bell would not be quite the same shape
insert into range_eg values (timestamp '2019-03-27 09:30:00');
insert into range_eg values (timestamp '2019-03-27 10:30:00');
insert into range_eg values (timestamp '2019-03-27 10:45:00');
insert into range_eg values (timestamp '2019-03-27 11:30:00');
--insert into range_eg values (timestamp '2019-03-27 11:45:00');
--insert into range_eg values (timestamp '2019-03-27 12:30:00');
--insert into range_eg values (timestamp '2019-03-27 12:45:00');
insert into range_eg values (timestamp '2019-03-27 13:30:00');
insert into range_eg values (timestamp '2019-03-27 13:45:00');
insert into range_eg values (timestamp '2019-03-27 14:30:00');
insert into range_eg values (timestamp '2019-03-27 15:30:00');
insert into range_eg values (timestamp '2019-03-27 16:30:00');

-- I added a small day at the end, to see how it may affect rolling total drift to zero
insert into range_eg values (timestamp '2019-03-28 11:30:00');
insert into range_eg values (timestamp '2019-03-28 12:30:00');
insert into range_eg values (timestamp '2019-03-28 13:30:00');

Then used an analytical function to cumulate the running count over a 1 day interval.
The second column, with accompanying union, was an attempt to see what happened if I filled out each hour with a record, to see if that influenced the overnight dip in my real data. It also let me see the cumulative count taper off as the data ended.

select * from (
    select rd 
     ,sum(sum(c)) over ( order by rd  
         range between NUMTODSINTERVAL(1,'day') preceding 
                   and current row)  last_24hr_running
     ,sum(count(*)) over ( order by rd  
         range between NUMTODSINTERVAL(1,'day') preceding 
                   and current row)  last_24hr_cnt
     from 
      (select 1 c,rd from range_eg 
      union all
      -- I even tried to pad out records to see 
      -- if the line would vary during the lulls
      select null c, trunc(sysdate,'hh24')+1-rownum/24
      from dual
      connect by level< 24*3
      )
    group by (rd) 
-- must ignore first day in inline query
-- as cumulative range will be skewed
) --where request_date > sysdate-2 
order by 2;

Data checks out, SQL checks out - does the graph check out?

Test case plotted

I think so, but given the context of the data, I don't think this line graph is appropriate, particularly around the period of no activity. More on that in the next post.

Filtering outliers from Oracle APEX activity logs

Last year I described a simple test case that described how to remove outliers from a fictional dataset using the STDDEV() analytical function .
http://www.grassroots-oracle.com/2017/06/removing-outliers-using-stddev.html

I want to follow this up with a practical case using one of my favourite data sets - the apex_workspace_activity_logs that record who opened what page, in what context, and how long it took to generate.

I've been keeping an eye on the performance of a particular page, after making a few performance adjustments to some conditions. Unfortunately, we had an unrelated anomaly that pushed average pages times quite high for a short period. Needless to say, this set of outliers transformed my beautiful performance indicating lines to a boxy bell curve.

Oracle APEX page performance data with extreme outlier

A great feature with Oracle JET is the ability to hide certain series, on click within the legend.
In this case I just wanted to ignore the MAX line for this post, which in this chart forms the secondary y-axis.

OracleJET Region Attributes - rescale

Our clients really enjoy this particular feature (so do I), so thanks to the JET team for building such a device, and the APEX team for integrating it.

This graph shows results where I modified the query to filter the outliers, on demand.

Performance graph with outlier removed

Looks like the adjustments to the conditions worked! The trend is downwards.

I tried a few variations to control the switch, but this seemed to perform the most predictably, although I'm not happy with the hardcoded number.

select [aggregate stuff]
from (
 select [all columns]
,case when :P23_IGNORE_OUTLIERS = 'Y' then
  -- only bother calculating when filtering them out
  stddev(elapsed_time) over (order by null)
else
  9999999
end as the_stddev
from [activity logs]
where [time/page is desired]
)
-- only when elapsed time less than 2 standard deviations gets 95% of your data
where (elapsed_time < 2*the_stddev )

I have a generic example of this on livesql.oracle.com I pay attention to these aggregates for our performance reports

• Median - what most users are experiencing
• Average - a typical user experience, influenced by extremes
• Moving average - general trend of visits, spread over a few days. An attempt to normalise local events
• Max - what's the worst some people are experiencing?

Here are some other activity log queries you may find interesting.

Happy graphing!

Modify OracleJET Property at Runtime in APEX

OracleJET has attributes galore, but some are are (not yet) available to change at design time, so JavaScript code can be added to the chart attributes to set relevant attributes.

function(options) {
options.styleDefaults.threeDEffect = "on";

return options;
}

See my previous post about modifying these attributes on render.
We can also do this at runtime, perhaps as response to a button click, such as the 2D/3D button in the cookbook.

First, set a static ID on the chart, possibly one of the most common "advanced" properties I use.

Static ID property available on many components

Use this ID in the browser console at runtime to see what JSON was generated for the chart definition.
Any missing properties will use the default specified in the documentation.
apex.widget.jet.getChartJSON('p95_skew')

Browser Console results

The documentation on these attributes is thorough, but I'd love some examples to help keep me moving.
http://www.oracle.com/webfolder/technetwork/jet/jsdocs/oj.ojChart.html#styleDefaults.threeDEffect

We can modify the orientation property by supplying a name/value pair as a JSON string.

$("#p95_skew_jet").ojChart({'orientation': 'horizontal'});

Note the selector has is the static ID with a suffix: $("the_static_id"+"_jet")

It took a little while to find the correct punctuation for the nested properties, so this is really one of those blog posts I created so I don't forget. You've seen Alex's blog by-line, right?

$("#p95_skew_jet").ojChart({'styleDefaults':{'threeDEffect': 'off'}});

Not all runtime tweaks behaved as expected, however. The following property behaves as expected when setting on render, but at runtime it squishes the width of the entire chart.
$("#jet1").ojChart({'styleDefaults':{'barGapRatio': 0.2}});

Refreshing the chart region afterwards did not help in this case.
While looking for answers, I came across this post from Riaz describing similar customisations.

OracleJET JavaScript Customisation in APEX

I've finally got some regular hands on a 5.1 instance, and the shiniest tool in the box for me is OracleJET.

Some months ago I spent a few days learning about OracleJET and the knockout framework with Chris Muir. I doubt I'd ever get to that nitty gritty, but it sure is handy to know some of the finer details now that I'm using them in APEX.

I wanted to have a play with the funnel chart. I took an existing query, and quickly busted out a working chart.

Upon playing with the sample chart in the OracleJET Cookbook, I decided I wanted the 3D option.
In the case of this chart style, I think the subtle effect made a big difference.

The Cookbook is a great guide to what you can play with

I couldn't find the relevant attribute defined declaratively in APEX, but that's fine - I know all of them aren't mapped, and we have a special JavaScript section in the chart attributes to help us customise the content. This expands upon attribute selections, not replacing them like the custom XML did for AnyCharts.

The attribute help in the Page Designer is a good start, and always worth checking when playing with a new field.

Page Designer Attribute Help

I noticed the Sample Charts application did the same thing, but I figured I could take this sample from the help, combined with information from the impressive JET documentation, and try work out the JavaScript myself.

Not all properties are mapped to APEX attributes

This is what I came up with.

function(options) {
  options.styleDefaults.threeDEffect = "on";
  
  return options;
}

Only to find the customisation from the sample application looked same same, but different.

function( options ){
    options.styleDefaults = {
        threeDEffect: "on" 
    };
    return options;
}

Turns out both are effective, once again demonstrating there's always a few different ways of doing the same thing in JavaScript.
I suspect the first option honours the object notation mentioned in the documentation, while the second assigns the value as a JSON name/value pair. I've seen similar behaviour in jQuery:

$('#item').css('color','blue')
vs
$('#item').css({'color':'blue'})

You can find other examples in your workspace with this query on the dictionary view.

select application_id, page_id, page_name, region_name, chart_type
 ,javascript_code
from apex_application_page_charts 
where javascript_code is not null;

You may also enjoy changing series colours with similar treatment by Colin Archer of Explorer UK.
http://www.explorer.uk.com/customising-chart-colours-apex-5-1/
The German APEX community also has a thorough rundown of JET charting in 5.1
https://apex.oracle.com/pls/apex/germancommunities/apexcommunity/tipp/5841/index-en.html

In my next post I'll explore how to play with these attributes at runtime, not just on render.

2016 Blog Review

It's February, so it must be time to do this. I explore what thoughts arise from looking back, and forward to the future. It helps me to remember stuff, decide what to do, and apparently some people find it interesting.

Tools

So many tools. And I arguably don't use enough of them.

APEX 5.1

It's out and I'm excited. Unfortunately, scheduling is super tight, so I'll be waiting at my current site until hopefully around June, but I've certainly been enjoying the play on apex.oracle.com. I look forward in particular to the OracleJET built-in charts, we've got some great dashboarding ideas to explore.

I've seen plenty of questions come through the forums on the Interactive Grid, but I'm by-passing most of those until I get the chance to use it. I sure know where it belongs, and I look forward to understanding how it ticks, and the implications it brings.

JET plug-in - work in progress

OracleJET Visualisations

After an interesting overview in the depths of OracleJET from Chris Muir, I assigned myself homework that's seen me quietly working away on some plugins, bringing OracleJET visualisation (charts) for use in 5.0. I'm not sure how successful the cultivation will become, but I am documenting my journey. Expect a decent series of articles on my process, but be patient. I've had a hiccup with JSON and I'm giving it a rest for a little bit.

12c

We have 12c at our main client site and I appreciate some of the features it brings, so much so I look forward to 12.2, particularly LISTAGG.

I particularly like Identity columns; the row limiting clause, though I probably abuse it sometimes; and I reckon I must spike page hits for Tim's post about lateral & outer apply joins.

I'm starting to explore performance benefits of PL/SQL in WITH clause, and the UDF pragma, among other features.

JSON

My colleague has done some interesting stuff with parsing incoming JSON using 12.1 SQL, and I'm slowly exploring with my plugin generation using APEX_JSON and LISTAGG. With made another step deeper into 12c by upping MAX_STRING_SIZE to allow for processing of larger JSON.

I'm quite glad it's no longer XML.

It seems 12.2 will be bringing the remainder of tools required to really get going with this. There's quite a lot of polish I've seen in this release that makes it an attractive upgrade.

Atom

After many, many years using Textpad, I'm giving other editors a go. Atom has shown good promise, but I'm still yet to get it compiling my packages on a Windows box, though I haven't persisted.
There are some other minor niggles, but it does bring great benefits. I might give Sublime a fair go this year, however. There's beer in it for anyone who can get me compiling from a text editor that pleases me. Sorry, I only use SQL Developer for ad-hoc queries, compiling from scripts, and some not enough built-in reports.

I am casually interested in how Atom was built. I see Android Instant apps as being something to keep an eye on with this amazing JavaScript tech.

JavaScript and node.js

I really haven't given node.js a fair go, and if there was anything other than OracleJET visualisations that I want to learn this year, it's node.js. Maybe my QNAP will help me learn after all.

Ubuntu

As much as I see the benefits of running a linux based operating system, I think I just need to accept my fate as Windows proficient, using Virtualbox when I can. Too many hurdles to transition operating systems when there is a world of user interface development to keep up with.

ORDS

I can't add much to this from this year, except we've had a steady keel for 12 months. I'd like to move to nearly whatever the current version is when we up to 5.1.
I did notice spikes in the connection pool that endangered other users when a page was opened containing an image gallery, particularly for a 'job' with a large number of associated images. We added pagination to that page to cap these spikes and keep them below our connection pool thresholds.

Community

Thanks to all those who answer and ask questions online (and at conferences, of course).

ACE

Plenty of hub-hub this year about the Oracle ACE program, and the potential for a number of Ace Alumni to appear. It's going through a maturation phase. Let's wait for it to evolve and continue to recognise and potentially aid those people helping the community thrive and develop. Kudos to all those diligent experts out there who remain unrecognised within this particular program, but support their teams and community with their humble expertise.

I'll continue to output stuff that helps me remember stuff and benefits other people, regardless of which side of the line I'll fall. Whatever helps my abstracts get submitted when I occasionally venture out in this giant planet ;p

Blogging

No real exciting stats to report this year. Growth minimal, but quite the regular heartbeat of visitors. Many of them me, looking up certain references. This is the very reason that tipped me over the edge when starting this blog, seeing this byline:

"Oracle Things I Got to Remember Not to Forget" - Alex

2016 Blog visitors - Lift your game, Greenland

For some reason, my reference to Carsten's LISTAGG function was the most popular this year. Perhaps because more people like us who are using this to generate larger JSON sets prior to 12.2?

Also up there for hits was about one of my favourite APEX development patterns.

It took until #22 in the most visits by page for a 2016 post, an important one on improving PL/SQL performance in APEX. Some of my favourites from this year include decommissioning triggers, and a debugging how-to that may be a useful reference in the forums (#40).

Spiking briefly was this Patterson-Gimlin style glance at APEX 5.2

APEX Sample Applications

If you haven't heard, Dick Dral possibly leads the pack at the moment in regard to sample applications, I need to have more of a play in there. He's been a busy boy!

While reviewing blog page hits I noticed activity this page where I started to catalogue my bookmarked list of community sample applications. Maybe I can finish modernising my sample app and make these more prominent, or find a better home elsewhere.

Book

For anyone who purchased a copy of my jQuery in APEX book, thank you, and I would recommend you re-source chapter 9 electronically. Somehow an early draft made it in but it has since been replaced, using what seems like a logistically amazing process in the world of publishing.

If you like video format, I did this video series for 4.x, but many principles still apply.

While I've got a back-log to read, this Real World SQL and PL/SQL is hard to go past. Well done, gang.

AI

Because I like science and technology.

Among other things, I predict forms of intelligent systems / AI will be a science we'll see more of in the world, solving problems that wouldn't immediately spring to mind. I also wonder if it will help clients become event thinner, so I can stop complaining about not having enough space for apps on my phone, even as a moderate user of apps.

Astronomy

Because I've liked this since I was single digits.

This year will mark the end of an amazing mission that I've been following for quite some time. Cassini has been exploring more than just Saturn as shown in this amazing timeline. It's already committed on its fateful descent into Saturn in September.

Saturn moon Iapetus

I know in next year's review I'll be talking about the amazing James Webb Space Telescope, scheduled for launch around a year after Cassini farewells. This thing is going to be the biggest thing for astronomy since Hubble. Literally. The few weeks between launch and confirmation of a functioning telescope will be the most thrilling moment that represents years of work for thousands of engineers. There's no second chance like Hubble had, nor any service missions, not while still humanity struggles to exceed low earth orbit.

Science, in general, is actually quite a big part of the current news cycles, but for all the wrong reasons. I wonder how this will change if China's Chang'e 5 makes it to the moon and back this year, making it the third country to achieve this amazing feat and the first in 40+ years. I wonder how POTUS will respond?

There is at least one reason to be in the land of the free this year, for on August 21 a total solar eclipse travels across the entire country, a rare event hopefully inspiring many budding young scientists.

Happy science, everyone.

APEX 5.1 Chart Column Mapping

If you've played with the D3 chart plugins you can find in the Sample Charts Packaged Application, you may have noticed the ability to nominate a column from your query as the series name.

APEX 5.1 Column Mapping

APEX 5.1 provides this built into the framework. Combined with some rather granular attribute control at series and axis levels, I think it will be harder to find tweaks that haven't been made declarative.

And if you do, there is a section in the chart attributes for JavaScript code. This also means if you do need to start hacking your chart, you're just adding to the existing attributes.

OracleJET Custom JavaScript

Compare this with the XML API for AnyChart 6.x where once you start using XML, say goodbye to most of your declarative attribute settings.

The ability to nominate a series column is optional, bringing more flexibilty to the SQL that no longer needs to conform to positional columns. I'm not sure yet if this will impact how often I use pivot/unpivot, but I'm sure it will help.

This post was brought to you by the result of last minute presentation screenshot prep and a prompt from this tweet, so thank Peter.

Charting Predictions, al a AskTom

The 'grand algorithm' favoured this particular tweet from Connor McDonald in my 'highlights'.

Use regression functions in Oracle to do basic forecasts https://t.co/gaKGHafJaQ

— Connor McDonald (@connor_mc_d) June 14, 2016

I found this intriguing considering a side project I've been tinkering on. The solution ended up looking much simpler than a model clause, though I'm going to need to let it digest for a while before I fully understand how it works. Maybe read the relevant documentation.

Trouble is, the final result isn't the sort of thing you want to print out on your dot matrix printer to show the big cheese, right? You want to graph that sucker.

So I create a page in Oracle APEX 5.1 (apexea.oracle.com) and created a line chart.

I split the results into two series using Connor's query within an inline view, in part because that was the fastest way I could think to get the graph plotted

select dte
 ,sz as actual
from (
with reg as (
    select regr_slope(sz, to_number(to_char(dte,'j'))) slope,
           regr_intercept(sz, to_number(to_char(dte,'j'))) intcpt,
           max(dte) nxt
  from t
  )
  select t.*, 'actual' tag
  from   t
  union all
  select nxt+r dte,
         trunc(intcpt+slope*to_number(to_char(nxt+r,'j')))
         ,'predicted'
  from reg,
       ( select rownum r
         from   dual
         connect by level <= 30 )
  order by 1
)
where tag ='actual'

The result was compelling.

A picture paints a thousand words

I also like the fact the declarative series name honoured, rather than a double quoted "Column Alias".

It will be interesting to see how this plots more varied data, considering the data variance issues Connor described.

I added a to_char(dte, 'DD Mon YYYY') to get a prettier tooltip, but interestingly, it changed the angle of the line.

TO_CHAR() added

Then I spotted this field here, I could use the query as supplied by Connor without having to think about pivoting the result to get it into the right format the engine requires.

A new declarative option

However the result didn't come out quite right. I think that's more an issue with the communication between APEX and OracleJET that anything else, ie: a bug.

Not quite right

It appears the declarative charting options have been engineered in 5.1 to reduce the amount of hoop jumping with the result set. Charting in general appears more declarative, easier to navigate and work out what's going on.

Nice work APEX team.