Search This Blog

Friday, 26 January 2018

What should we do with the Australian economy?

Matt Barrie’s recently wrote a very detailed analysis of the Australian economy which he published on Medium and LinkedIn. Summarising his points can’t possibly do an hour-long read justice, but here’s my best attempt:
  • Australia’s ability to create complex exports has dropped rapidly and is well below many un-technological third world nations. Our mining industry is barely relevant any more.
  • Our education system isn’t helping, with many students unaware of what an engineering degree offers and bizarre subject connections (e.g. computing and woodwork are in the same course for many students in NSW).
  • The most significant industry we have to speak of is preparing and selling real estate to Chinese millionaires. This foreign investment isn’t being well regulated in Australia, and is likely heavily driven by illegal currency transactions in China.
  • The effect of this is that it is becoming difficult for ordinary people to buy real estate, which in turn makes it harder to attract staff.
  • A cozy duopoly in our media makes it hard to have coherent conversations about these issues as they are only surviving the worldwide journalism train wreck because of real estate advertising revenue.
The challenge we have is coming up with effective solutions to these problems. It’s easy to call for reform. It’s easy to blame politicians for short-sighted decisions. It’s easy to complain and despair. But, what, specifically are the smallest and politically most-acceptable things we could do to address these problems?
I propose there are 9 micro-reforms that we could advocate for that have a reasonable chance of being enacted.

Earmark 0.1% of superannuation towards tech venture creation and funding

Startups rarely make money. Investors generally won’t make a good return on any startup. But we need there to be tech startups to keep all the rest of the economy innovative; it’s just that everyone wants someone else to pay for it.
So let’s mandate it. 0.1% of all superannuation funds need to be invested in tech startups. We’d need to have a definition of a tech startup, so I suggest that the definition should be this: an Australian company that qualifies for the R&D tax concession. While this doesn’t completely overlap with a “startup” (a young company optimised for growth), it’s pretty close. If a young company wants to grow, they will need to be breaking new ground and there’s likely to be some new knowledge being generated.

Expand R&D tax concessions to include open source

The R&D tax concession is the only effective government program that supports startups. QUT’s CAUSEE study showed that accessing a government program that was designed to help startups get going was one of the best predictors that a startup would fail.
The exception to this is the R&D tax concession and tax rebate — which is a strong predictor of success.
The idea behind these concessions is that if you are researching something — if you are generating new knowledge — then companies can claim between 125% and 150% deductions on the cost of doing this. Basically, if you have to perform an experiment to find out something, the Australian government rewards you for this because it’s increasing the stock of knowledge and skills in Australia.
Australia doesn’t have large software companies that can build their own stacks to do specialised tasks at the same scale that Google, Facebook, Microsoft and Amazon do. Atlassian and Freelancer aren’t in the same league. So to be competitive, we will need to co-operate.
There is already a model for cooperation among software development organisations: it’s called open source. If as a company you devote development resources to open source projects, you are providing the basis for cooperation with other Australian (and international) organisations, and you are adding to the stock of knowledge and capability in Australia.
So for all the same reasons that we should be supporting the tax concessions for research and development, we should be expanding them to cover effort involved in creating open source software.

Increase minimum wage for foreign technology workers

The USA recently did this for H1B visas. There were many problems with the 457 visa, but at the same time it did have some advantages.
On the downside, it was abused terribly by numerous multinationals. Since the company nominating a job role for a 457 role only had to demonstrate that they were paying the same wages as they were paying a local (and even then there were loopholes), all they needed was to promote one very junior Australian employee to a senior position at a low wage and then it was a free-for-all to employ foreign workers at the same seniority and (below market) wage.
On the other hand, Atlassian do pay fair wages to their local Australian staff and to their foreign workers on 457 visas, and it will be a major drag on their ability to compete in a worldwide market using staff in Sydney if they don’t have some avenue to hire overseas. Likewise Freelancer.
The solution doesn’t strike me as difficult. While the minimum threshold salary of ~$50k for foreign workers for some jobs where there is a worker shortage is reasonable, it’s absurdly too low for IT. Let’s set the minimum for an IT role (sysadmin, developer, analyst, salesperson) to $100k. That will let fresh graduates from our universities be able to compete for local jobs and get experience. Then, once they are experienced, it is fair for them to be competing for more senior roles in a world-wide marketplace.
A good programmer or sysadmin (local or foreign) can easily bring in $100k of value to their employer; one who can’t bring in that much value has no business being brought to Australia under a visa to provide “needed” skills.
If a foreign worker does have 5 years’ experience in something valuable, then we do them no justice by letting them be under-paid when they come to our shores for the first time with little idea of what their actual value add will be.
So: let’s have a minimum $100k wage for guest IT workers.

Track what happens to reports of immigration fraud

Visa fraud is being perpetuated by multinational IT companies. Not just Indian ones — although their fraud levels are legendary — but also house-hold names, well known companies who pretend to maintain a semblance of ethics. Every IT worker in Sydney or Melbourne can name some company that they have seen doing something way outside of the scope of the law.
Whether it is advertising for a “senior devops engineer” for $60k (about half the going rate) so that they can bring in a foreign worker to fill their shortages; or letting workers come for a “conference” where they happen to work 9–5 in their own cubicle for several weeks without meeting up with anyone in Australia; or bringing someone to help with “short term training to bring skills not found in Australia” (for classes like basic MS-Word training) — we all know it is happening.
And yet, as far as I can tell, these rampant crimes aren’t getting investigated. Among IT workers, we have become so inured to it that we barely bother reporting it. Nothing ever gets done: when was the last time you saw blatant visa fraud get prosecuted?
Reports to the Immigration Department get ignored as far as I can tell. You can submit a tip-off, provide as much evidence as you want (names, dates, what visa they arrived on, who their manager was and how the manager detailed to the employee precisely how to execute the visa fraud, why it is comically fraudulent etc. etc.). What you will get is a thank you note, and a comment to say that the Immigration department cannot comment on any ongoing investigation.
There’s a difference between commenting on the details of an investigation and commenting on whether anyone has bothered even to read the submission.
I’d settle for getting a reference number for an investigation and a notification of of where it is up to in the process (reviewed, dropped, continuing). Annual reporting about the number of investigations in each phase would be nice. I would imagine that the local Federal member would want a full and frank update about how many companies in their constituencies are under investigation as well.
This matters, because if we lose confidence in our country’s institutions’ integrity to do what they are supposed to be doing we will have bigger problems than what Matt Barrie outlined in his original write-up.

Require proof of residency to buy property

Australian property prices have gone crazy. A median income 30-year-old is now in a position to afford the bottom 10% of the property market (down from around 30% in the 1980s). That’s accounting for the low interest rates we have at the moment; one huge challenge for many younger families is saving anything after they’ve paid rent.
There are several causes, but high rates of immigration to the capital cities is a significant factor. On the streets of Shanghai and Beijing, it’s common knowledge that investing in real estate in Vancouver, London, San Francisco, Sydney and Melbourne is a safe bet.
There are more millionaires in China than there are houses in Sydney, so unsurprisingly, this has pushed the price of real estate in Sydney to absurd levels.
Amazingly, ASIC hasn’t even found a dozen cases of illegal real estate purchases by non-residents. Fascinating. It’s almost as if the system is deliberately set up so that it’s impossible to investigate.
So here’s my $0.10 on how to make Sydney prices more affordable for the next generation: as part of paying stamp duty on the purchase of a house, you should be required to show your residency status, and demonstrate that you have the right to buy it; otherwise the transaction is voided.
A photocopy of your passport or birth certificate will be sufficient paperwork for nearly any legal buyer; it wouldn’t add much in the way of burden.

Publish details of non-resident purchases

In fact, we barely need to add any resources to ASIC to make this enforceable. Let’s simply publish the real estate purchases of all foreign citizens. Even a busy Saturday could be recorded in the Bitcoin blockchain for a negligible expense and no effort by the land titles office — but we could equally well require the land titles office to run up a new system to publish it in a centralised database.
And why would that be sufficient? Because it shines a spotlight on mainland Chinese buyers who mysteriously can simultaneously buy a multi-million dollar property in Sydney while never once taking more than USD50k out of China. The Bank of China stopped publishing estimates of illegally expatriated money a few years back when it was over USD2 trillion; it’s probably much higher now. And with the current Chinese currency crisis (they will be out of foreign reserves in around 5–6 years) the BOC would really, really like to keep their capital controls in place. Easily-available evidence to help prosecute money laundering would really help a lot of people right now.
There are economic arguments that say that capital controls aren’t a good thing, but when the economists of the second largest economy in the world say that they want to do something, that really is a matter for China to handle internally and it is their sovereign right to do so. It’s not a human rights abuse. If China wants to make it illegal to take large amounts of money out of the country (as they do: the limit is USD50k per annum), Australia is under no moral or political obligation to protect Chinese citizens who choose to disobey their country’s laws. In fact, seeing such people prosecuted would be really very helpful for us.
So let’s have a big asset register of all foreign-owned property and let foreign law enforcement help solve our housing bubble for us.

Mandate solar panels on all buildings sold in high-value areas

Just hypothetically, suppose you had a house on a plot of land that was worth $500,000 and you wanted to sell it. Then suddenly the state government mandates that unless you have 10kVA of solar panels on your roof (which could be bought and installed for around $12,000), that they will fine you or alternatively, refuse to record the sale.
It’s possible that someone might want to buy your land at $512,000 after you’ve spent the money on the solar panels; but unless something has also changed in the economy it’s more likely that they will still only afford to pay $500,000.
So, you end up slightly worse off than you otherwise would have been. Effectively you only make $488,000, but in truth, if you are selling land in an Australian city, you probably are making a tidy profit anyway, none of which was anything other than inter-generational theft. This is a small tax on a large profit.
But the benefits are enormous and significant: 10kVA worth of solar panels on 9 million dwellings in Australia would produce around 130,000 gigawatt-hours of electricity per year. This is roughly half of Australia’s total power consumption; the 70 million tons of carbon dioxide emissions that this would save would meet our carbon goals for 2030 at no cost to any government.
The 1.8 million houses in Sydney would — during peak times — generate 18 gigawatts of electricity. This is substantially higher than the highest-ever peak load in NSW (which was approximately 15 gigawatts during the summer of 2010/2011).
Thousands of new jobs in the green economy would be created at the stroke of a pen.
So how about this? If the valuer general has valued a property at more than $200,000, the property can’t be sold without the capacity to generate 40kWH per day from solar sources. (That’s roughly what a 10kVA system should produce).

Increase target inflation rate to 5–7%

This isn’t something our government can do — interest rates are set by the Reserve Bank, and they operate independently. But Treasury can ask very nicely, and the Reserve Bank can give it some serious consideration.
Our politicians and econocrats have an irrational hatred of inflation, which is a pity because that’s what we need right now. We have below-target inflation, and only the hint that maybe perhaps some day we might get into an inflationary danger zone — and yet we hear talk about the next interest rate change definitely being a rate rise.
Let’s compare ourselves with Spain. Up until about 2008, Spain was doing OK, with good economic growth and a balanced budget. As it turns out, this was a bit of an illusion, it was mostly caused by wealthy Germans buying houses in Spain so that they could get some nice clear blue skies for a change. (Does this story sound familiar? Substitute “China” for “Germany” and “Australia” for “Spain” and that’s basically us.) After 2008, Germans started repatriating their money to pay off their debts and started complaining about the lazy Spaniards who couldn’t manufacture anything cost-effectively any more. (Still sounding familiar? What happened to Australian manufacturing?) This was of course because the Spanish economy was being dominated by real estate and gushing in-flows of money, so the manufacturing industry got destroyed. (Familiar again?)
Where things get particularly nasty is that the German Bundesbank has basically refused to lift their own inflation rate, which means to become cost-effective again, Spain has to have deflation. Now, while economists don’t like inflation, they have a total cold-sweat fear of deflation. That’s because we can’t control it, and it’s quite awful.
You are not likely to sell your house for less than you bought it. Generally you will stay in it and keep making repayments rather than take the on-paper loss. Likewise, you are not likely to leave your job if the only other option involves a pay cut. You will stay on in a job even though you might hate it. But Germany is asking Spain to lower wages and lower real estate prices. Let me spell out what that means: it means making people hate their jobs so much that they leave anyway; it means making companies who don’t fire all their staff go bankrupt; it means making people so economically desperate that they sell their houses with nowhere else to go.
Does that disgust you or frighten you? It should. Our only alternative to this (in Australia) is to inflate our way out of trouble. That means aiming to keep house prices steady (in Australia dollar prices — i.e. your typical suburban house should stay at around 1.5million), but have wages go up rapidly so that it becomes affordable again. We should be aiming for everything else (salaries, food, entertainment, etc.) to double in price over the next ten years while the dollar price of housing stays the same. We wouldn’t be as worried about buying a 1 million apartment if average wages were around $100,000 and a smashed avocado was $20.
Incidentally the consequence of this will be that the Australian dollar will become worth half what it is today; but that shouldn’t matter for anything else if we get this right.
Many economists will object to this plan, and will talk about NAIRU (non-accelerating inflation rate of unemployment). There’s a belief (which doesn’t have particularly strong evidence) that says that there’s a certain stable unemployment for a country and if you go above this that bad things happen. The idea is that if unemployment gets down to zero then the only way to get more workers is to lure them away from another employer with higher wages, which means you’ll have to charge more. This is inflation. Accelerating inflation is where everyone starts considering this as normal, and so to lure someone away from a job requires steadily bigger and bigger pay jumps. If the process isn’t stopped, eventually you end up with hyperinflation like Zimbabwe or the Wehrmacht.
It’s a reasonable theory, but as a tech geek looking at robotics, AI and biotech I think it’s hilarious. We are about to launch into the generation where robotics replaces manufacturing, where AI replaces large amounts of white collar work, where international outsourcing will be replacing most of the remainder of the white collar work that can’t be automated, and where our agricultural outputs are going to be dominated by how quickly we can splice new genes into plants, not on how quickly we can hire day labourers.
Runaway inflation is not how it’s going to play out in Australia in the future, and it’s not how it’s playing out right now.
Just looking at the tech scene it’s obvious that this isn’t happening. We’re hitting the limits of local tech employment without any doubt at all. As Matt Barrie pointed out, if he gets one good tech resume per day he’s happy. If we were going to start seeing runaway inflation, we would be seeing it in the tech scene now.
But we’re not seeing it — the latest economic reports show progression towards full employment but weak wage growth — because the link between getting stuff done and employing extra people has been broken.
As an employer with a need for something, the first step is never “let’s go and pay more and hire someone away from the competition”. The first step is always “can we automate this task away?” As robotics get cheaper and AI gets smarter the answer is going to be “yes” more and more often.
The second step anyway is “is there a company somewhere in the world that specialises in this?” A niche expert will always be better than an employee. I was talking to the founder of a company the other day — she specialises in fixing photographs in e-commerce stores to improve sales. She’s brilliant, and her company is growing rapidly because she can do this better (cheaper, faster, better results) than anyone can do it in-house.
The third step for most large Australian companies is “can we get this done by our Indian outsourcers?”; for smaller companies it’s the same question but the method is Freelancer. (Hey, Matt, did you like my plug for you?).
Only when there is really no choice does anyone put an ad up on Seek, and when they do they get deluged with people looking for 457 visa sponsorship transfers rather than salary-gain hoppers.
It is in this world that the Reserve Bank’s mandate is to make sure that Australia doesn’t have an over-employment problem. I can’t see this problem happening, so I can’t see why we need to keep inflation down. Outside of real estate, surely it’s the opposite. Let’s aim to keep house prices steady by whatever means possible — clear messages to all investors that Australian real estate is not going to be a good long-term investment by publishing their transactions and prosecution where required — and let a high inflation rate clean this mess up.
To paraphrase a quote from another era, let’s get Philip Lowe to credibly promise to be irresponsible for a while.

Publish numbers of children accelerated in schools

This might be a very strange policy suggestion, but hear me out on this. We have several problems in our education system in Australia (and NSW in particular):
  • We aren’t teaching the right things. When computing gets bundled into the same curriculum stream as woodworking, we will train up a generation of children to think that computers are all about design. This will leave them hopelessly underprepared for a world of AI, robots and software eating the world.
  • We aren’t delivering better education now than we were 20 years ago.
  • The system isn’t in any way fair: if you can afford to pay for tutoring, you can get your child into a selective school, regardless of your child’s actual abilities. Only one selective school in NSW matches the population for parental level of income and ethnicity. That school is not James Ruse, which is very much the school of the wealthiest quarter of the population.
  • We are spending way too much, particularly in the public system. When a suburban church with no educational administration experience can put together a low-fee-paying private school whose total running costs are lower than the public system but are competitive in outcomes with it, there’s something deeply wrong in the way our schools are spending their money.
  • We aren’t delivering an educational experience that makes foreigners want to come to Australia and have their children study here. (Unlike, say, Finland, or Germany, or to a lesser extent Singapore or Shanghai).
I would argue that they have as their root cause a general disinterest among our educational administrators in being scientific. There is a bias towards promoting humanities — and along with it, a bias against STEM. In NSW, you can complete your HSC without doing a single subject in science, computing or mathematics. But if you are a gifted mathematician who loves physics and engineering and programming (an easy 12 units which will directly help with your likely future career), sorry, you must take some English because without knowing Shakespeare you really can’t be let loose into society and be allowed to program a microcontroller.
I asked half a dozen schools whether one of my data science students could do a statistical analysis of the school’s Naplan results — to build a machine learning model of student skill growth; to answer “what are the actual underlying causes?” rather than just the draw pretty pictures of reported results. The report would be shared only with the school, and would let their English teachers pinpoint what areas were the most efficient to be work on in order to help every child pass at least band 8 in year 9. Most of them simply said no, that they were not interested. One was sort-of doing it, but for various reasons couldn’t take up the free offer. The one school that said yes happened to have a teacher who completed a degree in both maths and English, immediately saw why this would be useful and moved heaven and earth to get it happening. As for the others: being scientific, disciplined and being data-driven is a STEM thing that their humanities-driven administration aren’t interested in.
There are a few areas of educational research where we know the answers very well: where everyone who studies it comes out with a statistically strong signal. For example, we know full well (from dozens of studies) that if you want children to achieve fluency in a foreign language, that you teach them Esperanto first, get them fluent in that (which is much easier than any non-artificial language) and then teach the foreign language after that. Going from bilingual to trilingual is a very quick process; it’s going from monolingual to bilingual that’s hard — if you make mono-to-bilingual easy then the total time spent getting to three language fluency is less than if you try to jump straight to two language fluency using a native language. This is a well-known and easily reproducible result.
Of course, this has been completely ignored in educational policy across Australia, but at least there’s the argument that it’s hard to hire Esperanto-speaking teachers. (Actually, it’s not, but at least it’s a coherent argument.)
There is controversial research: computers add little to the classroom in non-IT classes except for distraction. I can understand why that sort of research hasn’t been brought back into the classroom yet.
Where we see life-changing, tragedy-inducing problems is when educational administrators ignore the 70 years of research on acceleration of gifted kids. Here’s the research summarised in two points:
  • If a child is gifted, then they should be accelerated unless the child has an older sibling in the year above, or unless the child doesn’t want to. This is the only way that an intelligent child can get a peer group that they can connect with, which is the only way to get social skills
  • If you don’t accelerate a gifted child, they have far worse life outcomes, problems of self-esteem and mental health and a variety of other issues.
I have seen administrators just state their opinions — completely at odds with the research — and are surprised when it is questioned. Doctors who continue to prescribe medicines that fail clinical trials and have bad side effects would be struck off the register, but educators can do it with impunity.
A school where you can ignore 70 years of research on a whim is not a school that is going to be taking advantage of the best practices in doing anything. It is not going to run efficiently; it is not going to improve.
So let’s publish statistics per school on the number of children that are being accelerated. A school with a thousand children should have around 20 Mensa-level full-year-accelerated children, and usually one child accelerated at least two years.
(Amusingly enough, this data is being collected in NSW schools already. School principals are supposed to report this to their regional coordinator. However, since the NSW Department of Education doesn’t think anyone would be in the least bit interested in this, and that it has no relevance to anything the department cares about or monitors, the regional coordinator is incentivised to throw the data away.)
Why does this help?
  • It will bring to light anti-science sentiment in schools, which at least gives us a chance to do something about it.
  • It’s a meaningful measure that smart parents can use to choose between schools
  • If — as a school — you have to acknowledge the existence of children who will rip through your curriculum in 4 years instead of 6, you’ll have to do something about this, rather than just ignoring it.
  • It’s just possible that if you know that you have a cohort of very bright, very mathematical children coming through your school that you can’t ignore, that perhaps you’ll work out some approach to keep them interested: that might involve STEM, biotech and meaningful computing studies.
  • If you are a smart expatriate with highly gifted children — exactly the kind of person we would want coming to Australia — you can get a sense of where you might find a like-minded cohort and social group for your children.

Make biotech and medtech the highest priority at CSIRO

As someone who grew up with computers, and who believes that being able to program is important to a child’s education as literacy is, it pains me to say it, but I think Australia has missed the boat when it comes to IT. I can’t see us having a programming workforce that can compete on a world stage; I can’t see us building data centres that are cost and power competitive with global needs; I can’t see us being the leading centre for robotic-driven manufacture.
It’s possible we will become the dominant economy when it comes to robotic mines.
Textile manufacturing is about to go through a dramatic change — now that robots are beginning to handle cloth and fabrics better, it will no longer make sense to manufacture in third-world sweatshops. It’s possible that woollen garments might be made closer to the source of the wool; so perhaps we might see a renaissance of (robotic) textile manufacturing in Australia.
But, these aren’t going to be sure winners.
Globally, Australia is good at making new medical devices. We’re in a race with Mexico to be the number one “best” location for doing that. The next decades of the twenty-first century will see a dramatic increase in the number of medical devices and their sophistication. That is, they will be physically much simpler, and easier to manufacture, but have more complex software.
Australia is also trusted for our ability to do biotech more broadly. When an Australian company releases a genetically modified wheat or rice or other plant, it’s viewed quite positively around the world. There’s a lot we could do: there are two forms of photosynthesis, one more efficient with water than the other. One way or another, with a drying climate Australian farmers will end up very knowledgeable about low-water crops, and will be stretching the boundaries of how little of various nutrients are required in order to grow a healthy crop.
Vat grown meat: there’s no reason Australia shouldn’t be known for this. It’s the kind of mass-scale high-tech that we can do. This will be the dominant source of protein for our children’s children, and will constitute an ever-growing part of our diets.
So: it seems that the most effective use of directed research and development funding is going to be medtech and biotech. Let’s see some heavy investment here, rather than the continual gutting of the CSIRO.
— — — — — — — — — — — — — — —
That’s the best list I can make. Feel free to add further ideas in the comments below: aim for initiatives that are win-win for everyone, achievable, only require small amounts of funding, and that are about directing the momentum of existing organisations in the right directions rather than trying to establish something new (which is always harder).
Also: print this article off, and forward it to your local members. Sign it with a hand-written note saying that you support these initiatives, and that you will vote accordingly. Put pressure on our politicians to make sure reform happens. Do it now, and set yourself a reminder in your calendar for 3 months, 6 months and 12 months from today to write again to your local members if you haven’t seen enough progress.

Thursday, 6 July 2017

Getting your fair share of AWS EC2, VMware, Hyper-V, etc.

One of the problems with virtualised infrastructure -- especially cloud servers -- is that your programs don't always get the CPU time that they need. The CPU time for your application is shared between CPU time given to other virtual machines on the same hardware. This is good for cost savings, but it would be nice to know how badly you are being affected by this.

I had a customer who had some really interesting VMware scheduling problems. They had a large number of multi-cpu virtual machines. VMware can't schedule a 4-cpu virtual machine unless there are 4 physical CPUs free. So even if you only have one tiny job to run on one CPU, VMware can't just schedule that one CPU -- it has to wait until at least 4 are available. (Incidentally, those other 3 CPUs that are scheduled but do nothing count towards co-stop% which is another interesting performance-and-tuning metric.)

As it turns out, almost all their virtual machines had some small tasks going on in the background (e.g. cluster heartbeats), so VMware tried very hard to schedule them all as best as it could, but the result wasn't pretty. How bad was it?

I wrote a little program that slept for a second, woke up, recorded the time, and then went back to sleep again. It kept statistics about how delayed it was. There were some horror stories --  there were virtual machines that received no CPU time for more than 90 seconds on occasions! No wonder their clusters kept crashing -- the cluster heartbeat time was only 30 seconds, so there was no way the cluster could stay up with VMware starving it of CPU time like that.

Anyway, I tidied up that program ("am-i-scheduled") and packaged it for RHEL7; the binary is so simple that it will also run on Ubuntu unchanged. I suspect it will run almost anywhere.

If you run AWS EC2 servers or other cloud-hosted servers, you really want to install this. This is the most convenient way you can find out how much CPU time you aren't getting: if your virtual machine is co-located with another virtual machine that is being used for Bitcoin mining, or password cracking, or a deep learning problem, you might want to terminate and try again. With am-i-scheduled, you can detect this easily, and measure the impact you are experiencing.

Here's the source: and the binaries (including RPMs) can be downloaded from here:

Tuesday, 16 May 2017

Two happy stories about the Australian legal system

Too often we whinge and whine about problems in the legal system. Let me share two stories where the legal system did what it was supposed to.
The first: a customer of mine wasn’t paying for work that obviously had been done -- a joint marketing program of Google adwords which were visible in the Google analytics console, a class I taught and some other hard-to-deny activities. There were all sorts of excuses and promises to pay by a certain date that would come and then go without anything happening. Months went by.
I’ve run IFOST for just under 20 years and I’ve tried to avoid court at all costs, even when I’ve been in the right and could have done so. It’s better to live at peace, forgive and forget and so on.
I don’t know why I finally flipped and decided that enough was enough. I threatened legal action, and didn’t get a sensible response. So I went to lawlink and created an account. Actually, I had to create three accounts: the first one was in the name “Greg Baker” but my passport, driver’s license and Medicare card all say “Gregory Baker” so I couldn’t validate my identity. The second I choose the wrong category of company, but I got it right the third time. About an hour wasted from some bad UI design on lawlink, but I’ve done it now.
Then I walked through an online form for a statement of claim. It was less than $10,000 so I was able to file in small claims court. It was straightforward enough to do so, but I wish there had been a better UI for calculating what the incurred interest expense was. You have to go to another page to get the annual interest rates for each year, turn that into a monthly interest rate by hand, apply those interest rates (e.g. in a spreadsheet) and then add up it all up. It could have been easier.
The filing fee was just under $200 and I received all the paperwork by email. I then forwarded the most official looking PDF to my contacts and said “you are going to have to answer this case, including paying the filing fees and interest.” I had an apology that afternoon from the most senior management at my client. I posted the statement of claim, signed a stat dec that I had done so. The day that their legal team was served with it, stuff happened and I was paid.
It was such a positive experience. The rule of law was respected, it wasn’t difficult or expensive or time consuming for me to make the case and the matter was resolved far faster than if I’d tried to keep chasing up the debt the way I normally do.
The second story: I was talking to my lawyer about the experience (she wasn’t involved in the case at all, I just thought I would let her know afterwards) and noticed that she’d written a piece about changes to contract law in Australia.
I am not a lawyer but as a small business owner dealing with multinationals all the time, all I can say is: wow. I have stacks of absurd contracts that I’ve had to sign to win business that have exposed me to a lot of risk. I’ve just relied on the good nature of my customers most of the time, and been burned occasionally. Or, sometimes, I’ve just walked away from good productive work that would benefit everyone simply because I couldn’t afford this risk.
But now I have some sort of protection: I can (and will) point out clauses that a court would disallow and leave it up to the customer to decide whether they want to keep the pointless clause in place.
In Australia, where the vast majority of the economy consists of small businesses, there really was no reason that our legal system should have allowed lopsided contracts, ever. This one small change to contract law will have a ripple effect through the Australian economy. I look forward to seeing how this plays out.

So that’s two bits of good news in a week.

Wednesday, 3 May 2017

Chatbots that get smarter at #AtlassianSummit

I’m in Barcelona at the moment, at AtlasCamp giving a talk about helpdesk chatbots that get smarter.

It’s easy to write a dumb chatbot. It’s much harder to write a smart one that responds sensibly to everything you ask it. Some famous examples: if a human mentions Harrison Ford, they a probably not talking about a car.
There are three different kinds of chatbot, and they are each progressively harder to get write.
  1. The simplest chatbots are just a convenient command-line interface: in Slack or Hipchat, there are usually “slash” commands. Developers will set up a program that wakes up to “/build” whenever it is entered into a room that pulls the latest sources out of git, compiles it and shows the output of the unit tests. Since this is a very narrow domain, it’s easy to get right, and as it is for the benefit of programmers, it’s always cost-effective to spend some programmer time improving anything that isn’t any good.
  2. The next simplest are ordering bots, that control the conversation by never letting the user deviate from the approved conversational path. If you are ordering a pizza, the bot can ask you questions about toppings and sizes until it has everything it needs. Essentially this is just a replacement for a web form with some fields, but in certain markets (e.g. China) where there are near-universal chat platforms this can be quite convenient.
  3. The hardest are bots that don’t get to control the conversation, and where the user might ask just about anything.
Support bots are examples of that last kind: users could ask the helpdesk just about anything, and the support bot needs to respond intelligently.
I did a quick survey and found at least 50 startups trying to write helpdesk bots of various kinds. It’s a lucrative market, because if you can even turn 10% of helpdesk calls into a chat with a bot, that can mean huge staff cost savings. I have a customer with over 150 full-time staff on their servicedesk -- there are millions of dollars of savings to be found.
Unfortunately, nearly every startup I’ve seen has completely failed to meet their objectives, and customers who are happy with their investments in chatbots are actually quite rare.
I’ve seen three traps:
  • Several startups have lacked the courage to believe in their own developers. There’s a belief that Microsoft, Facebook, Amazon, IBM and Google have all the answers, and that if we leverage or or lex or Watson or whatever they’ve produced this month that there’s just a simple “helpdesk knowledge and personality” to put on top of it, like icing on a cake. Fundamentally, this doesn’t work: for very soud commercial reasons the big players are working on technology for bots that replace web forms and with that bias comes a number of limiting assumptions.
  • A lot of startups (and larger companies) believe that if you just scrape enough data from the intranet -- analyse every article in Confluence for example -- that you will be able to provide exactly the right answer to the user. Others take this further and try to scrape public forums as well. This doesn’t work because firstly, users often can’t explain their problem very well, so there’s not enough information up front even to understand what the user wants; and secondly... have you actually read what IT people put into their knowledge repositories?
  • There are a lot of different things that can go wrong, and a lot of different ways to solve a problem. If you try to make your support chatbot fully autonomous, able to answer anything, you will burn through a lot of cash handling odd little corner cases that may never happen again.
The most promising approach I’ve seen was one taken by a startup that I was working with late last year. When they decided to head in another direction, I bought the source code back off them.
The key idea is this: if our support chatbot can’t answer every question -- as indeed it never will -- then there has to be a way for the chatbot to let a human being respond instead. If a human being does respond, then the chatbot should learn that that is how it should have responded. If the chatbot can learn, then we don’t need to do any up-front programming at all, we can just let the chatbot learn from past conversations. Or even have the chatbot be completely naive when it is first turned on.
The challenge is that in a support chat room, it’s often hard to disentangle what each answer from the support team is referring to. There are some techniques that I’ve implemented (e.g. disentangling based on temporal proximity, @ mentions and so on). A conversative approach is to have a separate bot training room where only cleanly prepared conversations happen. Taking this approach means that we substitute expensive highly-paid programmers writing code to handle conversations and replace them with an intern writing some text chats.
It’s actually not that hard to find an intern who just wants to spend all day hanging out in chat rooms.
Whatever approach you take, you will end up with a corpus of conversations: lots of examples of users asking something, getting a response from support, clarifying what they want, and then getting an answer.
Predicting the appropriate thing to say next becomes a machine learning problem: given a new, otherwise unseen data blob, predict which category it belongs to. The data blobs are all the things that have been said so far in the dialog, and the category is whatever it is that a human support desk agent is most likely to have said as a response.
There is a rich mine of research articles and a lot of well-understood best practice about how to do machine learning problems with natural language text. Good solutions have been found in support vector machines, LTSM architectures for deep neural networks, word2vec embedding of sentences.
It turns out that techniques from the 1960s work well enough that you can code up a solution in a few hours. I used a bag-of-words model combined with logistic regression and I get quite acceptable results. (At this point, almost any serious data scientist or AI guru should rightly be snickering in the background, but bear with me.)
The bag-of-words model says that when a user asks something, you can ignore the structure and grammar of what they’ve written and just focus on key words. If a user mentions “password” you probably don’t even need to know the rest of the sentence: you know what sort of support call this is. If they mention “Windows” the likely next response is almost always “have you tried rebooting it yet?”
If you speak a language with 70,000 different words (in all their variations, including acronyms), then each message you type in a chat gets turned into an array of 70,000 elements, most of which are zeroes, with a few ones in it corresponding to the words you happen to have used in that message.
It’s rare that the first thing a support agent says is the complete and total solution to a problem. So I added a “memory” for the user and the agent. What did the user say before the last thing that they said? I implemented this by exponential decay. If your “memory” vector was x and the last thing you said was y then when you say z I’ll update the memory vector to (x/2 + y/2). Then after your next message, it will become (x/4 + y/4 + z/2). Little by little the things you said a while ago become less important in predicting what comes next.
Combining this with logistic regression, essentially you assign a score for how strong each word is in each context as a predictor. The word “password” appearing in your last message would score highly for a response for a password reset, but the word “Windows” would be a very weak predictor for a response about a password reset. Seeing the word “Linux” even in your history would be a negative strength predictor for “have you tried rebooting it yet” because it would be very rare for a human being to have given that response.
You train the logistic regressor on your existing corpus of data, and it calculates the matrix of strengths. It’s a big matrix: 70,000 words in four different places (the last thing the user said, the last thing the support agent said, the user’s memory, and the support agent’s memory) gives you 280,000 columns, and each step of each dialog you train it on (which may be thousands of conversations) is a row.
But that’s OK, it’s a very sparse matrix and modern computers can train a logistic regressor on gigabytes of data without needing any special hardware. It’s a problem that has been well studied since at least the 1970s and there are plenty of libraries to implement it efficiently and well.
And that is all you have to do, to make a surprisingly successful chatbot. You can tweak how confident the chatbot needs to be before it speaks up (e.g. don’t say anything unless you are 95% confident that you will respond the way that a support agent will). You can dump out the matrix of strengths to see why the chatbot chose to give an answer when it gets it wrong. If it needs to learn something more or gets it wrong, you can just give it another example to work with.
It’s a much cheaper approach than hiring a team of developers and data scientists, it’s much safer than relying on any here-today-gone-tomorrow AI startup, and it’s easier to support than a system that calls web APIs run by a big name vendor.
If you come along to my talk on Friday you can see me put together the whole system on stage in under 45 minutes.