This is a follow-up to "Famous Last Words by Bosses I've Had" here: http://edweissman.com/famous-last-words-by-bosses-ive-had

Discussion of Part 1 here: http://news.ycombinator.com/item?id=3360862

I didn't make any of these up...

Boss: How long will the Shipping Module take?

Jim: 6 weeks.

Boss writes down "6 weeks".

Mark: But Jim, we can use ORP640 in the Shipping Module.

Jim: Oh right. 3 weeks.

Boss erases "6 weeks" and writes down "3 weeks".

Me: But ORP640 doesn't recognize intercompany orders. It'll take 3 weeks to add that functionality.

Jim: Right. 6 weeks.

Boss erases "3 weeks" and writes down "6 weeks".

Mark: But you don't need intercompany orders in the Shipping Module. We'll just make them transfers.

Jim: Right. 3 weeks.

Boss erases "6 weeks" and writes down "3 weeks".

Me: But we can't do intercompany transfers for military orders. We won't get the mil-spec paperwork. It'll take 3 weeks to add that functionality.

Jim: Right. 6 weeks.

Boss erases "3 weeks" and writes down "6 weeks".

Mark, Jim, and I bust out laughing. Boss checks his zipper.

Boss: We need the automatic scheduler to email us when the automatic scheduler is not running.

2 computers are next to each other in the Test Lab.

I switch the keyboards and mice, cover the wires with paper, sit down at the right computer, and turn off its monitor.

Boss sits down at the left computer.

Boss: OK, lets try this thing out.

Boss types, but nothing appears.

Boss: This isn't working.

I discretely type garbage, which displays on his screen.

Boss: OK, something's happening here.

Me: That shouldn't happen. What did you do?

Boss: I just tried to log in.

Me: Well, it didn't work. Try it again.

Boss types, but nothing appears.

Boss: It's still not working.

I discretely type garbage, which displays on his screen.

Me: What did you do? This worked perfectly in User Acceptance Testing yesterday.

Boss: I didn't do anything. I just tried to log in.

Me: Well, something's wrong. We'll have to look at it after lunch.

Boss: OK, but I'm going to have to add one more week to the schedule.

Boss, at the end of a 4 hour meeting to review new Time Sheet Codes: OK, that oughta wrap things up. Any questions?

Me: What code should we use for 4 hour meetings to review Time Sheet Codes?

Fred: "P809, Fucking Off".

Boss slams portfolio shut and storms out.

Laughter ensues.

Two days later, no more Fred.

Boss: We need a report that reports every bug the day before it happens.

Boss: Your expense report doesn't tally. It's off by 22 cents. I can't approve it.

One week later:

Me: The year end general ledger doesn't tally. It's off by $14,123.61.

Boss: Don't worry about it. We'll just write it off.

Boss: We don't have a cron in development. How will the nightly batch get done while we're testing it?

Me: I'll just keep a dev session open on my account and spawn a delayed job every night.

3 weeks later:

Boss: HR has informed me that you have too much idle time on your dev account. Why aren't you working harder?

Boss: Our Accounts Payable vendor is refusing to provide support until we pay our maintenance bill. Why aren't we paying them?

Me: Because the Accounts Payable software that we bought doesn't work and we don't have support.

A customer in South Carolina had been repeatedly reporting "deadly embraces" in their pessimistic locking application over the past 6 months. No one had been able to reproduce their problem.

Help Desk: Joe is calling from South Carolina. They're locked again.

Boss: I'm getting sick of this. Log off both sessions and close the ticket.

Me: Noooo! Don't do anything. I'll telnet into both sessions and find out what's causing this.

Boss: What should I do.

Me: Stay away from your phone and your computer for the next 20 minutes.

Boss: Sue did 32 tickets last month. You only did 7. Why don't you get as much work done as she does?

Me: I do. My tickets were for development projects. Hers were for bug fixes and password assignments.

Boss: So?

Boss: Sam pushed a bug into production last week. How do we prevent that from ever happening again?

Me: Don't let anyone do anything.

Boss: No, I'm serious.

Me: So am I.

Boss: Even since we started recycling paper and printing on the back side, I'm not getting the right reports.

Me: Look at the other side.

Boss: This customer in Michigan won't pay their bill. Why?

Me: They claim that the work they contracted for is incomplete.

Boss: Then why don't you finish it?

Me: Because the contract states, "...and anything else the client specifies until they're satisfied."

Boss: What idiot signed that contract?

Me: You did.

Boss: Fred in Buffalo loves you for speeding up his nightly batch by 90%. What did you do?

Me: I found a "SLEEP 10 SECONDS" inside a recursion. I removed it.

Boss: Stupid! You should have just changed it to "SLEEP 5 SECONDS" so we had more to give him next time he complains.

The company serves free bagels, donuts, and coffee to all of the programmers every Wednesday. Cost: $200.

Boss: In order to cut expenses, bagel day will now be every other Wednesday.

The company now serves free bagels, donuts, coffee, fresh fruit, juice, and egg sandwiches to all of the programmers every other Wednesday. Cost: $400.

Boss: You did great this year. I'm giving you a 2% increase.

Me: I hate you. I quit.

Boss: Then I'll give you a 4% increase.

Me: I hate you twice as much. I quit.

(OK, I made this last one up. But all the others are real, I swear)

Sorry to say, none of these were made up...

Me, 124 Monday lunches in a row: We need an adequate disaster recovery plan.

Boss: We do. We back up every day.

Me:   What happens when we try to restore one of those backups?

Boss: I don't know. Why?

Me:   Where's the test plan?

Boss: Jerry will make sure Fred's program works.

Me:   Where's the "Expected Results" section on the test plan?

Boss: What?

Me:   I don't have access to the production server.

Boss: I already emailed you your password.

Me:   I know, but I don't know my login.

Boss: What's a login?

Me:   That doesn't make any sense. Have the auditors approved it?

Boss: No, but we can't have everything.

Boss: I'm really upset that no one has updated me on Project 127.

Me:   I cc'd you on all 9 Project 127 emails I sent this week.

Boss: I haven't had time to get caught up on my email.

Me:   You've been invited to a meeting with 3 department heads to hash out their differences on Project 249.

Boss: I hate meetings.

Boss: Why haven't you started the Accounts Receivable project yet?

Me:   Because management has not yet decided whether customer credit limits should be per division or companywide.

Boss: What difference does that make?

Boss: We have hundreds of past due orders.

Me:   No, we have 22 past due orders.

Boss: I'm not going to argue with you.

Me:   Good, because you'd lose.

Me:   We're meeting with the customer at 8:00 a.m. tomorrow.

Boss: I hate mornings.

Me:   The server crashed. IT Services is working to bring it back up.

Boss: Don't confuse me with all these technical details.

Me:   The customer didn't receive that information because that product is not on our computer.

Boss: Give me a list of all products not on our computer.

Boss: Why haven't you started Project 193 yet?

Me:   Because the customer has not yet committed to the specs.

Boss: What difference does that make?

Me:   The program was written with 3 SQL selects inside a loop. It ran OK when we had 500 parts. Now that we have 10,000 parts, it runs real slow.

Boss: I don't understand.

Boss: What are you working on?

Me:   Project 432, which you said was my top priority. Remember?

Boss: No.

Boss: Why aren't you working on Project 387?

Me:   Because you said not to work on anything else until Project 432 was complete. Remember?

Boss: No.

Boss: I'm giving you only enhancements. I'm outsourcing all of the bug fixes.

Me:   But this is a bug fix. It says so right here on the ticket.

Boss: Oh, I didn't have time to read the ticket.

Boss: Amazon is threatening to shut us down because we ship too many orders late. How do we fix this?

Me:   Ship every order on time.

Boss: No, I meant, "How do we fix this with software?"

Boss, on December 31: Write a program to close every work order so we make our year end numbers.

Boss, on January 3:   Why is the database so screwed up?

Boss: You did great this year. I'm giving you a 2% increase.

Me:   I hate you. I quit.

Boss: Then I'll give you a 4% increase.

Me:   I still hate you. I still quit.

A client presented me with an obvious and significant problem that required immediate attention. I worked on the problem and helped them solve it. Along the way, I discovered a whole bunch of things that merit further examination by software developers.

The names and facts are changed and I will present the code as pseudo-code. I never compromise client confidences and the technology doesn’t matter: this could have happened anywhere.

This is pretty much bread-and-butter backroom application software for a large enterprise that processes lots of orders for lots of dollars...

I was presented of a pdf of a Purchase Order that had been emailed to a Vendor. The problem? No prices. Yikes. This could be a huge problem. The company emails thousands of Purchase Orders to Vendors every day, full of data supporting critical legal and mission critical transactions. The fundamental data elements are Part Number, Quantity, and Price. How could the price be missing? And how could it only be missing from one (or a few) out of thousands of Purchase Orders?

I started by doing what any digital sleuth would do: I tried to recreate the problem. Fortunately, this worked on the first try. I reprinted the Purchase Order and sure enough, no prices. I reprinted several others and there were prices.

The next step was to isolate the problem, debugging backwards. Output Record? Blank price. Variable feeding Output Record? Null value. Price on Purchase Order data base record. Fine. Hmmm. Next I examined the logic pulling the data from the data base and placing it in the output variable. It was looking for the Price in Column 22, the column for Foreign Currency Price. On an order to a California Vendor? OK, I was onto something.

I zeroed in on these two lines in the print program:

CurrencyCode = PORec[45]

if CurrencyCode = "USD" then PriceCol = 21 else PriceCol = 22

What was in Column 45 of this PO Record for this California Vendor? "USD" and a bunch of delimitters. Hmmm. That would cause PriceCol to be 22 when we obviously want it to be 21. The Price was in Column 21 but we are pulling a null out of Column 22. Bingo.

The customers are screaming. The business is suffering. Now what?

Stupid way out: Get the Currency Code from the Vendor record, not the PO Record

Lazy way out: Strip the delmitters from PORec[45].

Right way out: Find out what's putting delimitters into Column 45 of the PO Record.

Long term solution: See below.

The right way out can be very difficult with a large code base. First I isolated the 614 programs that had been promoted into production in the last 90 days. (I figured that the problem was new so the culprit program must be fresh.) I searched for the string "45". 42 hits. Nothing suspicious. Next I looked at data dictionaries and canned functions that provided potential synonyms for Column 45 of the PO Record. I found four possibilities. Then I searched the 614 programs for each of these. Nothing. Hmmm. Standards that no one follows. OK.

Then I simply scoured the list of 614 programs. One name caught my eye: "PoSplitter". Brand new. Written by a contractor who didn't know the whole application. Promoted 3 weeks ago. I read the whole program. No reference to "45", "Foreign Currency", or anything seemingly related. But one variable looked suspicious: DatasetCols. What was this? A list of columns in the PO Record that had matching multiple values, one for each Part on the PO. DatasetCols was a global variable passed down by a master routine. I read that routine and (bingo!) found 45 in the list of DatasetCols. I traced the mods back to 2005 when it was added to the list.

I double-checked the data dictionaries and the common functions. All said that Column 45 of the PO Record must be a single Foreign Currency Code defaulted from the Vendor Record and joined to a preset table. On the other hand, the master PO routine had it in a dataset list. A dataset list that had never been referenced by any other program until that contractor used it in PoSplitter. So, as soon as his program went into production, for every Purchase Order that was "split", Column 45 kept its original Foreign Currency Code along with a delimitter for each Part on the PO. Which in turn caused the PO Print program to fail to secure "USD" and automatically default to Foreign Currency (note that this bug would never affect foreign orders).

The immediate (right) solution:

1. Remove Column 45 from the variable "DatasetCols" in the master routine. Recompile all affected programs.

2. Clean up the data base.

The long term solution:

1. The data dictionary must be the Bible. Have no other code, variables, or function that can possibly say something else. Variables like "DatasetCols" must never be hardcoded, but must be populated from the data dictionary. All synonyms must also be defined in the data dictionary, not in many other routines.

2. Don't use datasets. Normalize your data. (Enough said).

3. Don't have hanging conditionals. Will If...then cover all possibilites? No? Then make a Case, catching any errors. ("USD***" is NOT a valid Foreign Currency Code!)

4. If something breaks, break it! The first time an error was encountered (see #3 above), the PO Print program should have stopped and demanded a help desk intervention. But since errors weren't being captured at the point of failure, 3500 Purchase Orders were printed without prices for three weeks before anybody who cared noticed.

5. Learn the app before you change it. I realize that this is easier said than done, but I'd like to think that the contractor should have understood what all the columns in the PO Record that he was changing. He simply trusted the variable "DatasetCols". Do you imagine that a senior developer would have caught that Column 45 was inconsistently documented in the existing code base? I don't know, but it's an interesting question.

6. Parallel test. The Split Line enhancement was big enough to run an automated parallel test. Column 45 of the POs from the test data base would not have matched those from the Control data base. This would have stuck out like a sore thumb if anyone had bothered to check.

7. Regression test. Just because the stuff that should have changed did change as expected, did everything that should not have changed stay the same? (I know, I know, how do we test for "everything else".) There's no easy answer for this, but doing nothing is the worst possible alternative.

What else would you add to my Long Term Solution list?