Thursday, May 31, 2007

Interviewing for DBAs

We are currently looking for contract DBAs and also a permanent full-time Senior DBA and I spent a good part of last week and this week reviewing resumes and conducting interviews.

Thankfully, there are a lot of interview questions out on the Internet. One of my senior DBA's favourite was "What was the most dangerous thing that you have done as a DBA?" and we get a wide ranging and variety of answers to that one. The good thing with this one is that there is no right or wrong answer but rather it provides us with an insight to the candidate's perspective of what is considered to be dangerous actions or actions that requires special attention and care.

We threw a few technical questions in the mix to allow us to gauge the level of technical expertise and knowledge of the candidates as we have found that 10 years spent creating database does not equate a senior level of knowledge and expertise as a DBA. Other questions are designed to identify the candidate's natural behaviour under stress or their thought processes in identifying and resolving issues.

One thing for sure is that it is practically impossible to judge simply from the resume but we still use the resume as our filtering mechanism on who should be interviewed and who shouldn't. I normally scan the resume for the type of work and organization that the candidate has worked for in the past where they picked up their DBA experience. For example, if someone put down that they spent 3 months at a small organization and was heavily involved in implementing RAC, 10gAS, DataGuard, replication, tuning, OEM, OID, RMAN, etc., then I got to question how much experience was picked up for each individual technology within the 3-month period. The other thing that I watched out for is whether someone spent the whole period doing DBA work or whether it was a combination of DBA, System Administration and/or Development work.

I can tell you that it is still a hit and miss exercise as one of my contract "hires" turned out to be a dud and I am still searching for the perfect process to hire the right person.

Wednesday, May 30, 2007

Sometimes tried and true methods do not work

You all are probably thinking "How could that be?". Well, it wouldn't work all the time because the action is on "people" who are as dynamic as anything.

The background to this was that a supervisor in another group sent out an email thanking a couple of her employees for their hard work and effort. One of my DBA was also heavily involved working with these guys on the same task and the email distribution list included him and a few others. My DBA was a little miffed and basically thought that he wasn't appreciated too. I told me not to worry too much about it and yes, I agreed that it doesn't help with the team work and team spirit that we were trying to encourage within the IS group. Guess what, the same thing happened again the very next business day. The same supervisor sent another email thanking another employee in her group for his hard work and effort on another task where another of my DBAs was involved and assisted greatly in tuning and resolving the troublesome query. My first reaction was to do a "reply-all" thanking my DBA and saying something like "I'm sure XXX didn't mean to exclude you from his appreciative e-mail." but I thought better of it and decided to speak to XXX about it the next day.

I spent a good portion of the evening thinking and comptemplating how to approach the subject without coming across as "angry" or even making the divisive between my group and XXX's group wider than it already is. One that I decided on was to go to XXX asking for his help in dealing with an issue. Oh boy, that didn't go well at all even when I told XXX that normally I wouldn't have bothered if it happened only once but in this case, it happened twice on two consective days. His reaction to the first email was that he wasn't aware that my DBA was also involved and admit that he might have missed my second DBA on the second email. Basically, XXX's reaction was that I should have sent out an email thanking my DBA for their effort even though I know that my DBA don't need me to tell them that they are doing a fabulous job handling the daily challenges.

Oh well, chalk that up to more bridge-building work to be done for me with XXX and his group.

Friday, May 25, 2007

Projects and Project Managers

It is really interesting to see that no two projects are alike even if the project objectives are the same. For example, implementing Oracle E-Business, depending on the organization and the project manager in question, things are done differently and even if things are done the same, the results might be totally different!

The challenge has always been to ensure that the project team understands and is committed to ensuring that the project deliverables are defined, planned, accepted and delivered on schedule and within budget.

A good part of project management is to identify and manage risks as the project progresses and a key component is good communication to and within team members. When a project manager strives to manage communication by acting as the "middle person" or the hub, then you can be sure that things are going horribly wrong. Now the project manager decides who should or shouldn't have the information and meetings are usually conducted with the incorrect audience or key personnel are missing. Another sign that your project might be in trouble is the project manager is overwhelmed but refuses to ask or accept assistance with a third sign being the project manager's focus was managing to the next milestones with no clear project plan other than what is being focused on.

For me personally, a successful project is one which meets its stated objectives, on time, within budget and the project team wants to work together again. Too many times, I've seen projects delivered on time, within budget but no one wants to work for the project manager or with each other again.

What options do you have when you are stuck on a project that seems to go nowhere? I would suggest talking to the project manager (it could be that you were put in the wrong role) first. If nothing else works then you are stuck and the probably recourse is to remove yourself from the project.

Why am I blogging about this? Well, our E-Business implementation project is in trouble (we already delayed the implementation twice already) and this time around, a detailed project plan was done with the help of all the team leads (testing, development, training, implementation, etc) BUT the project manager was not replaced and is still doing the three things that I mentioned above. He is managing communications by having the team leads reporting on progress on a daily basis and a weekly get-together to drill them on why tasks are delayed or not completed according to schedule. The problem with this is that proper focus was not given to tasks on the critical path. I would think that if a task is not on the critical path and has slack, I wouldn't worry about whether it is late according to the schedule but rather that if tasks on the critical path, then effort has to be spent to determine the impact and what can be done to resolve/reduce the impact to the project and schedule.

I've questioned and provided comments on some of the issues that I'd seen: planning wasn't completed as dependencies between each sub-teams have not been fully defined (e.g. system testing for a particular component starts before development for that component has finished); resources are overallocated (e.g. person A is shown to be busy on multiple tasks (100% for each task) on a given day). The response that I got was "What do I suggest should be done to resolve these issues?". Arghh!

Despite all these challenges, the folks who are doing the work are totally committed to the project, working really, really hard and lots of hours to ensure that things are done and completed and working correctly. That is one of the positive from this project. We are now about 12 to 13 weeks away from implementation and heavily into testing and resolving defects or identifying workarounds if we are unable to fix.

The plan is for my group to support the system after go-live and I am recommending that there be no "enhancements"/"changes" be made to the system for at least two months in order to ensure that it is stable. I am also recommending that the post-project review be handled by someone other than the project manager to ensure unbiased input and frank feedback from the project team members.

Saturday, May 19, 2007

20 years after "Man in Motion" tour

In a couple of days, it would be the 20th anniversary of Rick Hansen's "Man in Motion" world tour where Rick Hansen, a paraplegic in a wheelchair decided to go around the world in a wheelchair to raise money and awareness for spinal cord injuries research. I remembered the event as I was about to finish my final examination for my degree in Toronto. It was inspiring to see this 27-year man trapped in a wheelchair since 15 to decide that he needed to do something. It is no less inspiring than Terry Fox when he did his run across Canada to raise awareness and funds for cancer research.

Some facts from Rick Hansen's tour 20 years ago (source: Vancouver Sun - May 19th, 2007):
- 160 wheelchair tires worn out;
- 2 years on the road;
- 34 countries visited on 4 continents;
- 40,000+ km traveled;
- 8 hours average spent on wheeling;
- 1,000+ baseball caps received;
- 200,000+ letters of support; and
- robbed 4 times.

There are many awe-inspiring stores of the human spirit and let us hope that this is what will help humankind take the right steps in the right direction instead of boneheaded things like the Canadian Parliament's move of making a politician issue of questioning Hockey Canada's appointment of Shane Doan as the team captain for the IIHF World Hockey Championships or the fact that the politicians are able to quickly pass into law a bill giving themselves a 29% - 59% pay increase and a lucrative pension plan ($4.00 public money for every dollar contributed by the politician).

Friday, May 18, 2007

Global Knowledge 2007 IT Salary & Skills Report

Recently I got an email inviting me to download the 2007 IT Salary & Skills Report from Global Knowledge. Although the report did not indicate whether it is specific to a country/region, I have to assume that given that the company is headquartered in the US, that the report is specific to the US IT industry and that salary figures are in US dollars.

It is an interesting read but even though the report talked about certifications (e.g. CISCO, Microsoft, etc.), Oracle Certified Professional (OCP) was not mentioned at all. The average salaries were where I expected them to be with no big surprises except maybe for training where 31.2% indicated that their employer did not offer employer-paid training at all. For IT professionals, that's a death sentence meaning that you will have to pay out of your own pocket (after-tax dollars) to keep your skills current. I am guessing that a good number of this percentage are most likely independent consultants who basically have to pay for any training out of their own pockets but if you are an employee, then I would ask "Why would you work for an employer who has no interest or see no value in your skills and knowledge?".

Training is a major factor in the IT industry as technologies changes so much and so quickly that you will be obsolete if you can't keep your skills current. I have been in the IT industry since I graduated from college in 1985 and constant training through the years have provided me with skills and knowledge that has allowed me to progress and develop in the business.

Like any survey, the results are just gauges to be used to judge where you are in the spectrum but it does give you a good gauge of whether you are being compensated fairly for your skills, knowledge and experience.

Saturday, May 12, 2007

Posting a comment to Pete F's blog

I was attempting to post a comment on Pete Finnigan's blog when it comes back and showed me a preview of the post and a confirmation button to post it.

Clicking on that button gave me:

I tried it again and got the same thing. Maybe it's me ;)

I didn't know that! (or an obscure bug in

My guys at work came across a very obscure bug in one of our Oracle databases the other day. A SQL Loader script that has been working for a long while started failing with an ORA-03114 and the corresponding SQL Loader error was "SQL*Loader-941: Error during describe of table". A quick look at the script to verify that nothing has changed and that everything looks good with the correct username and password being used, etc.

A search of Metalink provided some links describing similar problems but not really related to what we were experiencing. Fortunately looking at the trace files generated in the user dump directory provided some clues Publishto where the error was occurred and searching Metalink on those gave as Metalink Note: 343199.1 (if the database was started up by other means than from SQL*Plus, the bug might occur) which at first glance seems to gives us pause as to "Huh?" and "We didn't do that..." reaction. A further clue in our log directories showed that the script had ran successfully a couple of days prior which provided a "missing link". Oh yeah, we had shutdown the database for a cold backup and restarted it using RMAN. The note indicated that the bug would occur intermittently when the database has been started up using other means other than from SQL*Plus. So we shut down the db and restarted it from SQL*Plus and tried the script again, it worked! Wow! so, what is different in the process of database startup when doing that from SQL*Plus compared to doing that from RMAN? Unless someone can compared the two code, I'm afraid that we will never know. The Note did said that the bug was fixed in so hopefully that "fix" is carried to 10g too.

It was a very good feeling that we were able to "fix" the issue.