The Event of the Week at Work

Happened around lunchtime on Friday.

First one of the platforms we use crashed HARD, requiring a cold reboot of the system that housed a whole lot of virtual machines. And of course my inbox was filled up with about 500 new messages alerting us to the fact.

But the other one was more intriguing – it was on the Linux based side. A provisioning script kept crashing at line 120 when it tried to launch Apache Ant. It would do a test and just puke and exit the script at that point. I said, and it was my first inlking – it was like Ant wasn’t in the path for the user we were running the script as at that point. Now we had to sudo su – in to do this and then change to another user using su – {username} which spawns a login session. But the PATH variable wasn’t being set with the Ant info.

So then I did the following looking for terms like “PATH”, etc. by popping up to the home directory and issuing the command

cat */.bash_history |grep PATH|su|sudo

What that does is goes through every user directory in the /home top level and search through every .bash_history file for those terms.

But oddly nothing turned up. I suspect someone made a change to the path on this system – but no evidence of it so it looks like someone edited their .bash_history. Which lead me to suggest something. Why not scrape the .bash_history files every minute looking for those terms and then log it into a database. Pretty easy to write the script to do this and just schedule a cron job to kick it off once every minute or two. So next time someone does something stupid – it’ll get caught and committed to a database.

The other thing I recommended is doing the following every now and then:

date >> ~/.bash_history

Which puts a time and date stamp into the .bash_history file. That way you can see WHEN something happened.

 

Advertisements

2 thoughts on “The Event of the Week at Work

    1. Well – this past Friday was a good day. I managed to help us figure out how to easily replicate a client to other VM’s so we could load balance.

      And I also finished up the script to cleanup older MySQL dumps. See, the way you do dumps in MySQL is mysqldump dbname – but they copy the actual INNODB files. So it takes a LOT of space. That’s why I had to write the scripts.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s