Search This Blog

Showing posts with label books. Show all posts
Showing posts with label books. Show all posts

Tuesday, 8 March 2016

The management spinoff of the space race and of the dotcom era


ISO9000 was -- to some extent -- the translation of the engineering practices that supported the space program in the 1960s into the world of business: keep your documents controlled, make sure you know that what you are doing is working, and so on.
The equivalent of 1960s space engineering in the modern day is software engineering, and the practices that have been developed by software teams will form the basis of best-practice management in another decade or two's time. 
What practices am I talk about? What is it that software teams do, that is so obviously right that it barely gets mentioned, but is so obviously lacking from non-software business management? (Put another way: non-software business managers drown in email to the point of unproductivity. Why is that? And what does github do which happens to solve these problems?)
  • Instructions, procedures, notes, tomes, meeting reports, decision summaries, discussions and so on are living documents. Programs are just an example of these documents. They need version control, and be dynamically generated into their final forms. Here's an article I wrote a few weeks ago about this: http://blog.ifost.org.au/2016/02/draining-meeting-bogs-and-how-not-to_16.html
  • Issues, bugs, requests, tasks and so on need to be tracked through a workflow, and assigned to teams and thence down to individuals. (Blog link: http://blog.ifost.org.au/2016/03/draining-meeting-bogs-and-how-not-to.html).
  • Operational support teams need to be accessible at short notice, and that often includes specialised development teams who are expensive to interrupt. This dichotomy is what drives teams to adopt text chat where one member of the team has the "disturbed" role to try to answer questions from outside teams. (Blog link: http://blog.ifost.org.au/2016/02/draining-meeting-bogs-and-how-not-to_23.html).
  • Outcomes should be expressed as tests that can pass or fail. For programs, this means automated test suites. Outside of software, this is generally "a well-worded contract" but will soon sometimes to mean "contracts encoded in a formal language and attested in the blockchain" or "contract terms that a robot lawyer can read and judge."
  • The longer and more continuous and automated the path is from the specification to the finished product the better. Automated builds, automated deploys, coding in domain-specific languages that let ideas be expressed simply -- these are what we use in software. The next decade will see the same ideas expressed in fashion (the dressmakers pattern gets passed to a robot textile workers), in manufacturing (the design gets 3D printed), in law (self-judging contracts) and so on. 
Anything else I should add to this list?
We always talk of the incredible failures of software engineering -- and there are many, and they can be enormous -- but rarely of the incredible progress we have made as well. The progress is invisible and assimilated; the failures become news.

Tuesday, 1 March 2016

Draining the meeting bogs and how not to suffer from email overload (part 4)


This is the fourth (and probably last) in my series of blog posts about how we unknowingly often let our IT determine how we communicate, and what to do about it.

Teams need to communicate at three different speeds: Tomes, Task Tracking and Information Ping-pong. When we don’t have the right IT support to for all three, things go wrong. This week I'm writing about Task Tracking.

I lose my luggage a lot. Domestic, international, first world, developing nations; I’ve had travel issues in more places than most people get to in their lives. I’ve even had airlines locate my lost luggage and then lose it again in the process of delivering it back to me.

Two of the more recent lost luggage events stood out by how they were handled:

  • On one occasion, a senior member of staff apologised to me in person, and promised that it he would have his staff on to it immediately; 
  • On the other, a bored probably-recent graduate took my details, mumbled the official airline apology and gave me a reference number in case I had a query about where it was up to.

I felt much more confident about the mumble from the junior than the eloquence of the senior.

Why?

Because there was a reference number. The reference number told me:

  • There was a process that was being followed. It might be a very ad-hoc process; it might often fail, but that's clearly better than the alternative.
  • If the team leader is micro-managing the tasks of their staff, it's because they are choosing to do so. The process must have been done a few times before, so the staff know what to do.
  • The team leader will probably not be the bottleneck on stuff getting done. We've all seen it, or been there: the project manager who is painfully overworked while team members are idle (often unknown to the project manager).
  • The process will therefore scale somewhat. Staff can be empowered enough that the process could scale without a single-person bottleneck.
  • It told me that there was a team of people who work on finding lost luggage, and that someone in that team would be working on it.
  • If a whole flight-load of luggage had been lost, scaling up resources to help wouldn't have been impossible.
  •  It didn’t matter to me who was working on it; if I was wondering whether any work had been done, I would have logged into their portal and looked it up, and wasted no-one’s time or effort but my own.

Having the manager’s name and assurance gave me no such confidence, but instead the sure knowledge that if I called the manager, the manager would have to get back to me (i.e. go and query the staff member he had assigned). This process would have consumed staff time and effort; which meant that if a large number of people had lost their luggage and were all asking the same question, that there would be gridlock and thrashing (so many interruptions that nothing gets completed).

In your team, who is busier? The team leader / project manager, or the technical staff doing the work?

If someone needs something from your team, how is that work tracked? Do they have someone senior that they call, or can they find out what they want to know wasting no-one's time but their own?

In a sensible, well-run organisation the staff with more seniority should have more idle time than the staff who report to them. Otherwise, they are the bottleneck holding back the organisation's efficiency.

If this is happening, then something is wrong with the way ticket tracking is being done.

The most common ticket software systems tend to be helpdesks or service desks because the scale of such organisations usually make it essential It is almost impossible to live without them in software development once the software has reached the complexity experienced in a very minimal viable product.

But ticket-tracking can be done with almost any technical team inside or outside of IT. Here's my list of the minimum requirements for a ticket-tracking system to be useful:

DUAL FACING Ticket tracking systems convey two important pieces of information: that something is being worked on (or that it isn’t) and what tasks people are working on.

On the one hand, the system needs to be easy for end-users to see where their request is up to which is why tracking tasks in a private spreadsheet doesn't work.

The ticketing system should automate messages to requesters, create follow-up surveys when the ticket is resolved, and so on.

On the other hand, the ticketing systems needs to be fast and efficient for staff to update, or else staff will batch their updates in a large chunk at the end of the day or the end of the week. It also needs to provide reporting to management to provide a high-level overview.

ENUMERATED You want discussions to be about "LTS-203" not "the project to update the website branding" so that everyone is on the same page. The tracking system has to provide a short code that you can say over the telephone or in a conversation, and that usually means some kind of short string of letters (perhaps three or four, or a pronounceable syllable) followed by a number. If that number is 7 digits long, you have a problem, because no-one will remember it, nor be able to say it.

EMBEDDABLE Whatever you are using for your tomes and reporting, you want to be able to embed your ticket information into it, and have the status there automatically update. This makes project meetings smoother and more efficient, because you can embed the ticket into the minutes and quickly glance back to last week’s meeting notes to see what needs to be reviewed. If software projects are involved, then being able to embed ticket references into the revision control system is very helpful.

UNIVERSAL If the entire organisation can work off the same ticket tracking system, that is ideal. One client I worked with had numerous $100,000+ projects being blocked -- unable to be completed -- because another (unrelated) department had delayed some logistics for efficiency. It required months of investigation to uncover -- which should have been possible to identify by clicking through inter-team task dependencies.

ADAPTABLE In order to be universal, the workflow needs to be customisable per team and often per project. For some teams, To-Do, In Progress and Done are sufficient to describe where the work is up to. For others, there can be a 20-step process involving multiple review points. ITIL projects often end up clunky and barely useable because the One Universal Process for all Incidents is really only necessary for a handful of teams, and the rest are forced to follow it.

LOCK-FREE When a project is urgent you will have more than one person updating a ticket at the same time. This isn't the 1980s any more: it's silly and inefficient for one user to lock a ticket and leave someone else idle, unable to write. Time gets lost, and more often than not, the update gets lost as well.

PREDICTIVE We live in an era of deep-dive data mining. It's no longer acceptable to say to a customer or user "we have no idea how long this is going to take" any more than an advertising company could say "we don't know how much money to spend on your campaign". And yet, I still see helpdesks logging tickets and giving no indication to their user base of when to expect the ticket to be resolved. At the very least, make sure your ticketing system uses my three rules of thumb. Or better still, make sure it works with my Automated Estimator of Effort and Duration to get the best real-time predictions.




That seems to be the most important seven criteria.

  • Spreadsheets don't work, and yet I still see them used in almost every organisation.
  • Most startups use Jira or Basecamp. Basecamp also includes capabilities for Tomes and Information Ping Pong. 
  • Best Practical's RT is the most mature of the open source tools (even though it doesn't meet some criteria). Trac is another commonly-used open source tool, particularly when it is bundled with software repository hosting.
  • Large enterprises often use HPE Service Manager (which is less popular than it was in the past and is being replaced by Service Anywhere), ServiceNow (who took a lot of HPE's marketshare) and BMC Remedy. They are generally 10x - 100x the price of Jira but are designed to work better in highly siloed organisations.

Be aware that if the nature of someone’s job is that they will work on the same problem for months or years -- for example, a scientific researcher -- there’s probably little value in ticket tracking because there would be so little information to put in there. Likewise, if someone’s job is to do hundreds of tasks per day, then any ticket tracking will have to be automated by inference from the actions or else the overhead of the tracking system might make it impractical.




I'm hoping to put this (and many other thoughts) together in a book (current working title: "Bimodal, Trimodal, Devops and Tossing it over the Fence: a better practices guide to supporting software") -- sign up for updates about the book here:  http://eepurl.com/bMYBC5

If you like this series -- and care about making organisations run better with better tools -- you'll probably find my automated estimator of effort and duration very interesting.

Greg Baker (gregb@ifost.org.au) is a consultant, author, developer and start-up advisor. His recent projects include a plug-in for Jira Service Desk which lets helpdesk staff tell their users how long a task will take and a wet-weather information system for school sports.

Tuesday, 23 February 2016

Draining the meeting bogs and how not to suffer from email overload (part 3)

This is the third in my series of blog posts about how we unknowingly often let our IT determine how we communicate, and what to do about it.

Teams need to communicate at three different speeds: Tomes, Task Tracking and Information Ping-pong. When we don't have the right IT support to for all three, things go wrong.

In this post, I'll talk about team communication Information Ping-Pong.  Information Ping-Pong is that rapid communication that makes your job efficient. You ask the expert a question and you get an answer back immediately because that's their area, and they know all about it.

It's great: you can stay in the flow and get much, much more done. It's the grail of an efficient organisation; using the assembled team of experts to their full.

Unfortunately, what I see in most organisation is that they try to use email for this.

It doesn't work.

Occasionally the expert might respond to you quite quickly, but there can be long delays for no obvious reason to you. You can't see what they are doing -- are they handling a dozen other queries at the same time? And worse: it is just contributing to everyone's over-full inbox.

The only alternative in most organisations is to prepare a list of questions, and call a meeting with the relevant expert. This works better than email, but it's hard to schedule a 5 minute meeting if that's all you need. Often the bottom half of the list of prepared questions don't make sense in the light of the answers to the first half, and the blocked-out time is simply wasted.

The solution which has worked quite well for many organisations is text-chat, but there are four very important requirements for this to work well.

GROUP FIRST Text chats should be sent to virtual rooms; messages shouldn't be directed to an individual. If you are initiating a text-chat to an individual, you are duplicating all the problems of email overload, but also expecting to have priority to interrupt the recipient.

DISTURBED ROLE There needs to be a standard alias (traditionally called "disturbed") for every room. Typically one person gets assigned the "disturbed" role for the day for each team and they will attempt to respond on behalf of the whole team. This leaves the rest of the team free to get on with their work, but still gives the instant-access-to-an-expert that so deeply helps the rest of the organisation. (Large, important teams might need two or more people acting in the disturbed role at a time. )

HISTORY The history of the room should be accessible. This lets non-team members lurk and save on asking the same question that has already been answered three times that day.

BOT-READY Make sure the robots are talking, and plan for them to be listening. If a job is completed, or some event has occurred, or any other "news" can be automatically sent to a room, get a robot integrated into your text chat tool to send it. This saves wasted time for the person performing the "disturbed" role.

Most text chat tools also have "slash" commands or other ways of directing a question or instruction to a robot. These are evolving into tools that understand natural language and will be one of the most significant and disruptive changes to the way we "do business" over the next ten years.


Skype and Lotus Notes don't do a very good job on any of the requirements listed above. Consumer products (such as WhatsApp) are almost perfectly designed to do the opposite of what's required. WeChat (common in China) stands slightly above in that at least it has an API for bots.

The up-and-coming text chat tool is a program called "Slack", although Atlassian's Hipchat is a little more mature and is better integrated with the Atlassian suite of Confluence and Jira.

Unlike most of the tools I've written about in this series, the choice of text chat tool really has to be done at a company level. It is difficult for a team leader or individual contributor to drive the adoption from the grassroots up; generally it's an IT decision about which tool to use, and then a culture change (from top management) to push its usage. Fortunately, these text chat tools are extraordinarily cheap (the most expensive I've seen is $2 per month per user), and most have some kind of free plan that is quite adequate. Also, there's a good chance that a software development group will already be using Hipchat, which means that adoption can grow organically from a starting base.

Outside of a few startups, text-chat is very rare. And also outside of a few startups, everything takes far longer than you expect it to and inter-team communication is painfully slow. It's not a co-incidence. We think this mess is normal, but it's just driven by the software we use to intermediate our communications.




The next post in the series will hopefully be next Tuesday.

I'm hoping to put this (and many other thoughts) together in a book (current working title: "Bimodal, Trimodal, Devops and Tossing it over the Fence: a better practices guide to supporting software") -- sign up for updates about the book here:  http://eepurl.com/bMYBC5

If you like this series -- and care about making organisations run better with better tools -- you'll probably find my automated estimator of effort and duration very interesting.


Greg Baker (
gregb@ifost.org.auis a consultant, author, developer and start-up advisor. His recent projects include a plug-in for Jira Service Desk which lets helpdesk staff tell their users how long a task will take and a wet-weather information system for school sports.

Tuesday, 16 February 2016

Draining the meeting bogs and how not to suffer from email overload (part 2)

This is the second in my series of blog posts about how we unknowingly often let our IT determine how we communicate, and what to do about it.

Teams need to communicate at three different speeds: Tomes, Task Tracking and Information Ping-pong. When we don't have the right IT support to for all three, things go wrong.

In this post, I'll talk about team communication by Tomes.

Tomes are documents that answer "Who are we? What do we do? How did we get here? Why are we doing this?"

These are often important things to say, but they aren't particularly urgent. They might or might not be read by people inside the team. Quite often they are read by people outside of the team, but often weeks or months after they were written.

Examples include:
  • Project decisions
  • Strategic plans
  • Frequently asked questions
  • Status reports for long, ongoing projects 
  • Minutes of meetings
  • Discussions about any of the above

These are often tedious to write, and the task is often given to the person with the most free time using whatever tools are at hand. Generally, this means either email or MS-Word documents. There are serious flaws with both of these. Writing emails just contributes to the email overload problem. Writing a Word document and putting into a fileshare (or on to Sharepoint) leads to more meetings because it won't get read. Emailing a Word document around (which is probably the most common approach) manages to be the worst of both worlds.

I've made an attempt at distilling the 5 requirements for a tool to support Tomes. If you can think of more, let me know.

NAME MENTIONING If someone is mentioned in a Tome they need to be notified automatically. This keeps everyone on the same page (virtually). A common miscommunication I've seen is where everyone was expecting a response or involvement from someone outside of the team -- but the individual was never aware that they were supposed to be doing anything. Automatic notification mitigates this.

TASK ASSIGNMENT AND DUE If a task is mentioned in a Tome, it should be easy to assign it to someone and to give it a due-by date. The task should automatically appear on that person's to-do list. Otherwise, it is too easy for the task to get lost.

LIVE TASK STATUS When someone marks off a task as completed from their to-do list, this should be reflected automatically in the Tome. If a task has a lifecycle (e.g. it has to be reviewed by another team before it can be marked off as "Done"), then the live status of the task should appear.

This saves time in meetings working out whether something has been done or not. One of the most frustrating meetings I can remember (and I've had a few) is was a pointless nothing-to-report meeting: everyone admitted in the meeting that they hadn't had time to do anything and that there was nothing to report. This could have been trivially seen by the project manager who convened the meeting if he had been able to see the lack of progress on the task list of the minutes of the previous meeting.

IN-PAGE COMMENTS Comments about the Tome should appear with the tome. It should be easy to discuss and have a conversation about any part of the Tome.

CHANGE TRACKING AND WATCHING It should be easy to change the Tome. If it is changed, then it should be easy to see what changed even though the Tome displays the latest version by default. It should be possible to register an interest in changes to the Tome and get notified when this happens.



These seem obvious, and seems like a reasonably complete list.

  • Email can do #1 (you can CC someone outside your team). If you are using Outlook you can drag the email into your todo list (and Gmail can do something roughly equivalent), so that satisfies #2. But #3 and #5 are impossible, and #4 by email is a recipe for giving everyone a full inbox that just gets dumped.
  • Word documents in a portal or fileshare do #1 very poorly; #2 is even worse. #3 is impossible, but does OK on #4 or #5.
  • Slightly more innovative companies will use Google Docs. The collaborators list can act as #1, and it does as well on #4 and #5 as a Word document does. #2 and #3 are sort-of possible if the tasks are put into a Google Sheets document instead, but it is rather fragile, complex to set up and means that there are multiple documents for the one event or topic.
  • Startups often use Confluence. It nails all 5 requirements very well.



So what usually happens?

In large, traditional organisations, I see teams with Word documents being emailed around by a project manager, a deluge of emails (CC'ed to everyone the project manager sent the project tracking document to) and then meetings at least once a week so that everyone can work out whether or not they were supposed to have done something, and why that probably wasn't a high priority anyway.

Smaller organisations manage better, although the ones that try to do this with Google Docs tend to be very chaotic, and have someone always complaining about how they really haven't got their procedures and processes sorted out yet.

We think this mess is normal, but it's just driven by the software we use to intermediate our communications.

The next post in the series will hopefully be next Tuesday.

I'm hoping to put this (and many other thoughts) together in a book (current working title: "Bimodal, Trimodal, Devops and Tossing it over the Fence: a better practices guide to supporting software") -- sign up for updates about the book here:  http://eepurl.com/bMYBC5

If you like this series -- and care about making organisations run better with better tools -- you'll probably find my automated estimator of effort and duration very interesting.


Greg Baker (
gregb@ifost.org.auis a consultant, author, developer and start-up advisor. His recent projects include a plug-in for Jira Service Desk which lets helpdesk staff tell their users how long a task will take and a wet-weather information system for school sports.

Tuesday, 9 February 2016

Draining the meeting bogs and how not to suffer from email overload (part 1)


Some meetings are important; sometimes face-to-face is the best way to work through an issue. And email is a necessary business tool. But in many of the organisations I work with, I've seen meetings and emails used as a crutch because their staff aren't given what they need in order to work more efficiently.

I blame IT for this, perhaps too harshly, but IT should be thinking both about how individuals communicate, and also about the requirements for teams to communicate.

In general, there are three broad ways that teams communicate:
  • With Tomes that answer "Who are we? What do we do? How did we get here? Why are we doing this?"
  • Using different ways to say "We're working on it"
  • By playing Information Ping-pong


To be efficient, it's important that staff from outside the team can "lurk" (watch what is going on) without engaging the team across all three methods.

If other staff can't lurk -- they will either email you or ask for a meeting.

What I see all too often is desperate staff, who are over-worked because they are forced to use email and meetings -- tools which are very ill-suited to all three speeds of communication.

I'll discuss each of them in follow-up blog posts; I'll schedule them for Tuesday each week unless something else more interesting crops up.

I'm hoping to put this together in a book (current working title: "Bimodal, Trimodal, Devops and Tossing it over the Fence: a better practices guide to supporting software") -- sign up for updates about the book here:  http://eepurl.com/bMYBC5

If you like this series -- and care about making organisations run better with better tools -- you'll probably find my automated estimator of effort and duration very interesting.


Greg Baker (gregb@ifost.org.auis a consultant, author, developer and start-up advisor. His recent projects include a plug-in for Jira Service Desk which lets helpdesk staff tell their users how long a task will take and a wet-weather information system for school sports.

Wednesday, 23 December 2015

Please write a review

To everyone who downloaded a copy of any of my Data Protector books last week when I put them on special -- thank you! It was a very nice birthday surprise to take out #3, #4 and #5 positions simultaneously in Amazon's Business Software books.

If you found any of those books helpful, don't forget to leave a review: even just leaving a number-of-stars rating helps me work out what to focus on.

And if you missed out, you can buy them (they aren't very expensive even when they aren't on special) at http://www.ifost.org.au/books

Thursday, 10 December 2015

Celebratory 100th blog post - what people have actually been reading, and what freebies and junkets are on offer

This is the 100th blog post, and the counter of page views is about to tick over 50,000. Thank you for your readership!

According to Google's infallible stats counters, here's what most people have been reading on this blog:

POEMS At the end of a day of being a computer nerd, you need something that will make you laugh (or at least smile) and also make you look cultured among your friends. Nobody else writes poetry about nuclear physics or time travel, so if you want to get that "I'm so hip" feel, you really should buy a copy of When Medusa went on Chatroulette for $3 (more or less, depending on your country of origin).

NAVIGATOR - If you run Data Protector then you will definitely get some value out of the cloud-hosted Navigator trial. You can get reports like "what virtual machines are not getting backed up?" and "how big will my backups be next year?" -- stuff that makes you look like the storage genius guru (which you probably are anyway, but this just makes it easier to prove it).

HOW LONG WILL IT TAKE - If you are tracking your work in Atlassian's fabulous JIRA task tracking system, then try out my free plug-in (x.ifost.org.au/aeed) which can predict how long tasks will take to complete. And if you are not using JIRA, then convince everyone to throw out whatever you are using and switch to JIRA because it's an order of magnitude cheaper, and also easier to support.

TRAINING COURSES - You can now buy training online from store.data-protector.net -- and it appears that it's 10-20% cheaper than buying from HPE directly in most countries. There are options for instructor-led, self-paced, over-the-internet and e-learning modules.

SUPPORT CONTRACTS -  Just email your support contract before its renewal to gregb@ifost.org.au and I'll look at it and figure out a way to make it cheaper for you.

BOOKS - If you are just learning Data Protector, then buy one of my books on Data Protector (available in Kindle, PDF and hardback). They are all under $10; you can hide them in an expense report and no-one will ever know.

Friday, 13 November 2015

My CMDB book is inappropriate

I was contacted by Apress, who wanted to publish A Better Practices Guide for Populating a CMDB. But after their editorial team reviewed it, it was deemed "inappropriate".

So if I've offended anyone because of my inappropriate writing, please accept my apologies.

Anyway, nearly two years later it's still sitting at #70 on Amazon.com.au in its category, so even if Apress don't want to sell it -- and you are interested in buying a highly inappropriate book (PDF, Kindle, etc.), then have a look at http://www.ifost.org.au/books which has links to everything I've written. Here's the spiel:
This guide is an in-depth look at what you should and should not include as configuration items in an IT Services model. Unlike many other books on this topic, this goes into deep technical detail and provides many examples. The first section covers some useful approaches for starting to populate a CMDB with high-level services.
  • What is the least amount of work you can do and still have a valid CMDB? 
  • What are some techniques that you can use to identify what business and technical services you should include? 
  • Can anything be automated to work more efficiently?
The second section covers three common in-house architectures:
  • LAMP stack applications
  • Modern enterprise web applications
  • Relational databases, with some brief notes about other forms of database
The final section details how to model applications delivered through the cloud, and what CI attributes can be useful to record. This section covers the three main types of cloud-delivered application:

  • Software as a service applications, using Google Apps as an example.
  • Platform as a service applications, using Google App Engine as an example
  • Infrastructure as a service applications, using the Amazon and HP infrastructure.











Thursday, 16 July 2015

Data Protector books in PDF format for Europe; also it's now possible to buy them (anywhere) using Bitcoin

Apparently folks in Europe having been having problems buying my books through Distribly; presumably this is related to EU VAT.

As an experiment I have set up a breathtakingly ugly Shopify store for Data Protector books which should work for anyone, anywhere. It accepts Paypal (and hopefully therefore European credit cards as well). It also accepts Bitcoin -- contact me for a discount code to get 50% off while I test this capability.

The books I've put there are:

I'm not expecting anyone will be interested in the old book on Data Protector 8 (you should buy one of the v9 books instead as it will be almost entirely applicable). Also, I haven't heard of any problems for anyone wanting to buy A Better Practices Guide for Populating a CMDB in PDF from Europe, so I haven't put that on the store either. (Let me know if you need it.)

The links to buy for Kindle, on paper or through Distribly are unchanged: http://www.ifost.org.au/books

Greg Baker is an independent consultant who happens to do a lot of work on HP DataProtector. He is the author of the only published books on HP Data Protector (http://www.ifost.org.au/books/#dp). He works with HP and HP partner companies to solve the hardest big-data problems (especially around backup). See more at IFOST's DataProtector pages at http://www.ifost.org.au/dataprotector

Friday, 6 March 2015

A thank you to all my readers - Amazon's 4th best-selling book on disaster recovery

I checked on Amazon's best sellers list for Disaster Recovery books tonight and discovered mostly that there are quite a few mis-filed books there (e.g. chef is not about disaster recovery!)

But the top four that actually are about disaster recovery are:

  • Michael Lucas' book Tarsnap - an encrypted storage system for Unix users who want extreme levels of paranoia about the security of their backups. Michael is a very famous sysadmin-writer who deserves first place for anything he writes.
  • Steven Nelson's book from 2011 which covers enterprise backup in general, glaringly omitting HP Data Protector.
  • Joe Kissell's book on Crashplan
  • My Planning, Deploying and Installing Data Protector 9.
Thank you to everyone who has bought it so far; I hope you have found it useful.

Greg Baker is one of the world's leading experts on HP Data Protector. His consulting services are athttp://www.ifost.org.au/dataprotector. He has written numerous books (see http://www.ifost.org.au/books) on it, and on other topics. His other interests are startup management, applications of automated image and text analysis and niche software development.

Tuesday, 3 March 2015

Suggestions and updates for Data Protector books - a big thank-you to Chris Bhatt

Chris contacted me today having found numerous typos and corrections. They will propagate through the system over the next few days: Kindle users should see these update silently, I can send an updated PDF if you ping me and future print copy orders through CreateSpace will be the corrected version. Contact me if you want an errata list.

Chris also pointed out a number of things that I should have put in, but didn't. So here goes:


  • When you are writing to a virtual tape library (VTL) that does any kind of de-duplication, set the concurrency to 1. The reason you want to do this is so that you get very close to the same stream of data from one backup to the next: this will maximise your de-duplication. But there are some exceptions:
    • If you are using mhvtl on a Linux machine to emulate a tape library, then it doesn't make any difference what you do with concurrency. It does compression but that's only for very short streams of symbols that have already been seen before in that session.
    • If you are using a StoreOnce box, then don't use the virtual tape library option. Use Catalyst stores (StoreOnce devices) instead, as they use space more efficiently, and keep track of what has expired (and free it up). If you are worried about performance and want to do this over fibrechannel, this is possible if you are on Data Protector 9.02 and a suitable firmware version (3.12, for example).
  • I should have written more on whether servers should each have their own backup specification or one backup specification containing multiple servers. I think I'll do a blog post on this. When I've written it, it will be http://blog.ifost.org.au/2015/03/when-should-i-put-all-my-servers-into.html
  • It's worth reminding everyone of Stewart McLeods StoreOnce best practices (http://h30507.www3.hp.com/t5/Technical-Support-Services-Blog/DPTIPS-Data-Protector-StoreOnce-Software-Information-and-Best/ba-p/180146#.VOX1SXlyYdV).  I had a customer just last week who lost their system.db file -- but they could easily have lost a store.db as well -- because of some disk corruption. I will ask try to expand out these and a few other suggestions in the next edition.
  • Another question that deserves an answer is "what's the best way to backup and restore files on a volume that has many thousands / millions of small files?". In version 7.x and before, this was a major bug bear because the database was so very slow that it could become the bottleneck. Often the only option was to turn off logging altogether on that backup object. Even today it's worth splitting it up into smaller objects (which I mention in chapter 6 -- look for Performance - Multiple readers in the index). I've also since realised that I never quite described exactly how the incremental algorithm works with multiple readers either.
What else should I add? What have I missed? What would have helped you when you started out?

Put any comments below (as blog comments, or on Google+) or email me (gregb@ifost.org.au) with your suggestions.

Greg Baker is one of the world's leading experts on HP Data Protector. His consulting services are at http://www.ifost.org.au/dataprotector. He has written numerous books (see http://www.ifost.org.au/books) on it, and on other topics. His other interests are startup management, applications of automated image and text analysis and niche software development.

Tuesday, 24 February 2015

Planning, Deploying and Installing Data Protector 9: For the datacentre, the cloud and remote offices



This is the fourth book I've published on HP Data Protector, and I think it will be a very valuable resource to any sysadmin or consultant working with HP products.

If you are involved in pre-sales or implementation consulting you will find lots of useful material in this book.

  • Why customers choose Data Protector, and what its strengths and weaknesses are.
  • Step-by-step, everything you would need in order to backup remote branch offices.
  • Very detailed designs on how you can backup Amazon-hosted cloud servers.
  • How to backup data centres filled with nearly identical virtual machines.
  • Guidance on handling media pools, tape drives and other physical devices.
  • Doing disk-to-disk-to-tape backups.
  • Very detailed information on the internal database that simply isn't available elsewhere -- the results of my reverse engineering what HP has done. If you have complicated reporting needs, this is the book to have.
If you are a system administrator / backup administrator, you will find this book a welcome alternative to the HP documentation. I talk honestly about bugs that I've found, what makes sense to do, and what doesn't, and more importantly why certain designs make more sense than others.

This book can also be used as a course book if you want to run a one-day training session. All the labs can be run up in the cloud.

To keep the price down on the paperback version, it's a B&W print for all 374 pages. The Kindle version is full-colour if you have a full-colour device.

The paperback version has a comprehensive index to make this into an ideal reference to have at your desk. It is enrolled in matchbook, so you can have both the paper version and the kindle version for only slightly more than the paper version. Buy both!

Topics covered:

  • Reasons customers choose Data Protector 
  • Areas of comparative weakness 
  • Installing Windows and Linux cell managers 
  • Agent deployment methods 
  • StoreOnce de-duplication 
  • Backing up filesystems 
  • Backing up VMware 
  • Backing up cloud-hosted servers 
  • Configuring tape drives 
  • Managing media 
  • Backups spooled via disk 
  • Remote office backups replicated to a central office 
  • Reporting 

There is also a companion book on Data Protector for operators with a topics around restoring, supporting and maintaining.

Buy it on Amazon ( http://www.amazon.com/dp/B00T68FR64 ), or if you want to use an alternate store, they are listed here: http://www.ifost.org.au/books

Greg Baker is one of the world's leading experts on HP Data Protector. His consulting services are at http://www.ifost.org.au/dataprotector. He has written numerous books (see http://www.ifost.org.au/books) on it, and on other topics. His other interests are startup management, applications of automated image and text analysis and niche software development.

Thursday, 19 February 2015

Operating, running and supporting HP Data Protector 9

A new (Chinese) year, and I've written two new books on Data Protector.

The first is a book for operators.

  • If you are a sysadmin, and a new staff member who has just joined your organisation and needs to keep an eye on the backups. Buy this book for them, it has everything they need to get started.
  • If you are a consultant and you have just finished an implementation, you will need to provide some hand-over documentation. Buy this book for them, and you might only need to write a page or two of site-specific documentation. Everything else they will need is in this operators book.
There's an ebook version if you need something very cheap in a hurry; and there is also a beautifully presented full-colour on-paper version, which is more expensive, but makes a big impact. 



Topics include:
  • What a cell manager is; what an installation server is
  • How to install a Data Protector console
  • Using Manager.exe and what different connection error messages mean
  • Monitoring backup sessions
  • Reviewing backup history
  • Responding to day-to-day issues (mount requests, errors, tape busy)
  • Formatting tapes
  • Logging a support call with HP and collecting debug logs
  • Restoring filesystems
  • Restoring sessions
  • Restoring individual files
  • Restoring virtual machines
  • Restoring individual files from virtual machines using the VMware granular recovery extension

The paperback version also includes:
  • A page where you can record the name of the cell manager, where the cell console is, and any installation servers.
  • An extensive index
The e-book is available for pre-order today (with delivery next week), the dead tree version should be available by Monday. As with everything else I have written, you can find it a http://www.ifost.org.au/books

Greg Baker is one of the world's leading experts on HP Data Protector. His consulting services are at http://www.ifost.org.au/dataprotector. He has written numerous books (see http://www.ifost.org.au/books) on it, and on other topics. His other interests are startup management, applications of automated image and text analysis and niche software development.