Noon: I can start updating again.� Someday I'll write about why I�had to stop for so long.
- Dirty Girl &�I will be moving into a new house this week.� Two move events, Friday-Sun, and then Monday the movers come to get the big/heavy stuff
*********
8AM - Must finish SuperSearch systems Edison & Tesla by Friday 8/28.
9AM - WAS�Upgrade meating with Peter &�the rest of the upgrade team.� They need both NWIE�VM�images set up by Friday also.
Since 10am this morning I've been working on ITP project, ME2:
- ITP ME2 servers are moved to Columbus
- Cuervo and I�are re-ip'ing them and trying to get them to start up normally in the new network space.� Stuffer was being a total pain in the arse for a while but I�think I�have him on-board with the action items.
- must edit /etc/ldap.conf &�/etc/openldap.conf to add line:
nns_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus,satellite
- edit /etc/modules.conf &�remove reference to pcnet32
- edit /etc/fstab and remove 3 lines added by vmware
- After demter &�hestia are done, the rest of the boxes should come up without issue
- Forgot to change /etc/resolv.conf.� Hestia can't resolve anyone so she's not letting us in.� but that shouldn't happen with the nss line.� hmm.
- chkconfig off rhnsd &�osad?
4:40PM - every one of these servers are fucked.� none of them can get on the network to talk with their LDAP host, so none of them will let anyone log in.� What a fucking nightmare.� I�have to break into every one of these machines and fix the configurations.
All of them get host name changes
IP�addresses changed
Resolvers/Search
LDAP hosts
VMWare Tools
Loghosts
NTP�host
Mail Relays
Backup Clients
This has to happen or it won't come online.� If it won't come online, you can't log in.� :)
9:10PM
- Mr.�T said that the firewall rules were just finished and the network is shut off on these servers until the IPs are changed etc.� No wonder we couldn't even ping gateway.� It would have been nice to be informed of that bit of trivia.
- fuckers -
- We're going to call it a day and sleep a bit if we can.
- AM�- must talk to Cole and get my access level bumped up so I�can initiate tool installs without hand holding.
�- Have to get nets turned on and test accesses.
�- Get Cuervo the info that he needs to carry this on without me, or with Stuffer.�
- Dirty Girl &�I are going to meet Nikita to give her the $$ and get the $K.
�- I'll meet Grandpa Simpson and trade my Mercedes for his Suburban for the weekend.� Dirty Girl takes Scooby to work, I�bring Suburban back here and finish Oracle server build for WAS Project, and get UID's created on Edison &�Tesla for Country Boy, Becky, and Kipper.
*******
Calling it a night, logging out of work.� Going to start tearing down the other computers and prepping them for the move.
******
As a human being with (perhaps above?) average intelligence, I am of course concerned about the economic issues facing the Nation in which I live, and the Nation that I love.
I have been a reader of history and economics for a while.� Not because of my job, but because it interests me.� I'm an Engineer, and my natural talents lie in problem solving, problem resolution, and what we call "root cause analysis".� That is to say that when a problem is discovered on a computer system or network, it is my job to not only fix the problem, but to figure out what caused the problem and to prevent it from ever happening again in the future.� To that end, I say this to you:
The first step in finding any solution, is to identify the problem.
Every day I pay attention to what people in our Government say and how people in our Government are approaching this issue that has been fabricated by the Banking Cartels on C-SPAN.� Our inept "Leaders" can't even identify the problem, and they are proposing a solution that is bad for This Nation and bad for We The People!
First, read just the first section of this bailout VERY closely, and try to follow along with me:
Title I - Troubled Assets Relief Program:
Sec. 101. Purchases of Troubled Assets
[Skip to section]
(e) Preventing Unjust Enrichment. - In Making Purchases under the authority of the Act, the Secretary [Paulson - Former CEO of Goldman Sachs...Can you say 'PART OF THE PROBLEM?'] shall take such steps as may be necessary to prevent unjust enrichment of financial institutions participating in a program established under this section, including by preventing the sale of a troubled asset to the Secretary at a higher price than what the seller paid ot purchase the asset.� This subsection does not apply to troubled assed acquired in a merger or acquisition, or a purchase of assets from a financial institution in conservatorship or receivership, or that has initiated bankruptcy proceedings under title 11, United States Code.
So let me get this straight.
The (privately owned) Federal Reserve Bank (with Assistance from the Congress and President of the United States), kept interest rates artificially low and encouraged a wild speculation/real-estate bubble by lending money to people who could not afford it.
You people.� You know who you are.� You bastards who make $35,000USD/yr and think you're on top of the world.� Charging everything on credit cards, driving a $45,000 Infinity and living in a $500,000 McMansion, who are losing your jobs and unable to pay your bills.� You're the cause of this problem.� You and the Banking Cartels.
That means YOU should be the people to suffer for it.� Not the rest of us.
What this section ALONE, (this is just the first fucking section), what this section does is this:
You bought an over-priced McMansion on a $500,000 loan.
You can't pay for it, so the Lender who financed it is forced to seize the asset.
Now the the Lender has this asset and they have to sell it.� But they have a LOT of assets and nobody is left to buy the assets at the inflated price.
So they have to sell it for less, and thus they lose money on the deal and thus fail because they have lost entirely too much at this point.
Now when they fail, someone bigger like JP Morgan will swoop in and buy up those Assets at "firesale price".� I.E. Pennies on the Dollar.�
Well now the Congress has given the (privately owned) Federal Reserve Bank the authority to put on the tab of We The people, any mortgage or essentially ANYTHING that they deem necessary to keep a bank from failing...
And if what is being charged was gotten through a merger....(purchased for pennies on the dollar), that the ailing bank can dump it on the TAXPAYER for the amount of what the IDIOT borrowed or even MORE.
So they pretend to stop them from making excess profits, yet give them implicit authority to do JUST THAT in the next goddamn sentence.
Why is anyone still listening to these assclowns?
Very hot schedule today, stand by:
9:10 - turnstone shut down and memory pushed down to 1024mb.� pulled out lm-tools and installed gcc.� Must add VMWare tools and recompile.
- Kipper is going to get the LM-QA-Zodiac image up-to-speed and we'll spin a copy out to physical (wabasha if there is room)
- Check out turnstone with 1gb of ram and see if performance is still out of whack
- Build on towhee will continue today only.� That will get TOmcat up to speed, KIpper and I will get WAS up to speed today.
- Make decision today about which to use, Tomcat or WAS.� We could stick with the LM install of WAS, then do a stand-alone build in a few months.
----
- BTI meeting right now.
- Need to deploy servers.
- Witty Jim is working on the DB builds/installs.� He's dorking with DB2 now.
- Ratface wants 3 seperate servers stood up for this build.� They can be fresh 4 builds or fresh 5.2 builds.� They can be clones of genseed.� He doesn't care.� We just need fresh boxes with WAS 6.1 on them.
-� Will probably house with PFF.
10:30
-- Talked with Mr. T about this clusterfuck and have a few changes:
- BTI will house with PFF, but not only on the same ESX hosts, but on the same VM.� We need to share BTI, PFF, and K&K all on the same VM and same WAS instance.�
- Makes perfect sense to consolidate and use fewer WAS licenses
- Tosses a little more complication into the mix because I have 3 apps contending for the same resources.� (And the QA WAS install is fucko)
::
Slick is pulling the Zodiac image out of LM for deployment on Wabasha.
Once he finishes, he'll boot Zodiac up and let Kipper know.
2:15 - Turnstone re-deploy and re-conversion is complete.� Turned it over to Kipper to check out the WAS install with 5GB.
- Spoke with Mr. T. as 1-1 meeting.� Have meeting with AVP IceQueen tomorrow.
-- found out that smmsp user can't be deleted or sendmail won't run.� fixed that.� also fixed it in the RH5 gold image.
Amy M. wants a CMS system pronto.
They want to use juno to smbclient mount various shares from windows servers so they can copy the content from windows into a linux box. (juno)
Then use juno to spray the files into place every month.� Six files, once a month.� They don't want to do it manually anymore.
Grandpa Simpson is going to upgrade Augur to 3.0.2 today.� We'll deploy a 5.2 Gold image out to augur and check firewall rules to see how we can get the data moved out.
Investigate private VLAN for this.
---
Kipper was having problems with turnstone so I raised memory to 5120 and restarted.� They rolled into swap briefly but it seems to be doing a little better now.
Ratface needs his password reset on genseed.� Need to find out which iteration/IP he's working on.
--
Have to pull RHEL4GOLD into LM.� I copied a machine called mvalent out to cilantro, but I have to talk with Slick about getting it ready for the mvalent install.
LDAP testing this morning at 7:30.� I pulled stats and put them into a file.� Haven't sent them to Hoops yet.
Welcome back!� I wish I felt better, but really, I don't.� So lets just get into it straight away shall we?
9:30 - Immediate start on LDAP troubleshooting.� How long have we been doing this?
I have windows up into mandrake, goblin, atlas, poseidon, dionysus, eros, demeter, and hestia.� We're cranking logging up to the tits and getting ready to sniff all traffic as we blast users through.�
It looks like the LDAP server isn't returning the request properly.
Bridge line is open. 09:45
%%%% Telstar rebuild is on hold.
%%% Red Hat 5.2 image is DONE!
1:55 - The ldap testing is finished and Hoops is going to analyze the data.
- Took my bicycle out for a spin to Fry's at lunch.� Picked up a few things I wanted (salad, soup, and some soda) and rode back.� Took about 45 minutes round trip, then a shower afterward.
- Starting on the Telstar satellite replacement now.� Will probably be on this the rest of the day.
- Rammstein wants to know if anyone took leela down last night between 23:00 and 02:00.� I did not, and I let him know that.
- He is working with Chops and Premie on fixing the WICK apps.� I'm copied on the emails but am not getting involved unless I see them doing something excessively weird.
9:11 - Discussed the issues on Clover & basil with Slick.� Sent Staph-o an email asking him to check them out with Monitor.
- Yesterday Prince Chili Longhorn and I had a difference of opinion about the PFF QA/Dev envs.� He wants to leave them in Lab Mangler and I was under the impression that the servers would be spun out to carson & wabasha and renamed.� They'd be set up to assume the identity of the current presentation layer.� He says that's not the case.
- Apparently he wants to leave the QA - WAS migration machines in LM and proceed to install Prod using WAS instead of Tomcat.� He says Mr. T. is onboard with that.�
- I think its a bad idea because the Prod environment would be built with a completely different design and layout from Dev and SWT.� I don't think this fits into our ESDM model.
10:15 - I wanted to start working on the RHEL5 gold image today.� VIC in New York didn't give me a 5 option until I moved over to Mage.� Mage has 3.02 on it, Seer only 3.01.� 5 Wasn't an option on 3.01.
- Kipper needs me to look @ gargoyle.
� - A few weeks ago Kipper called saying that he couldn't get processes restarted after a change window on cyclops & gargoyle.� It can't spawn enough processes he says.
2008-08-26-09:46:05.742-07:00I----- 0x38AD51C2 webseald ERROR wiv pthread WsWorker.cpp 495 0xb5c06080 DPWIV0450E Could not create new thread (11) 2008-08-26-09:46:05.814-07:00I----- 0x38AD50A4 webseald WARNING wiv general IVServer.cpp 978 0xb5c06080 DPWIV0164W Could not start background process
Explanation: WebSEAL failed to create an additional thread. This may be due to running out of operating system resources or exceeding process limits. Action: Check memory and thread limits for the process, and available memory. The error number can be looked up in /usr/include/sys/errno.h for more details on the problem.
- I raised the nproc limit in /etc/security/limits.conf a few weeks ago, set the hard limit up from 1000 to 3000.� Today I set the soft limit up from 400 to 1000.� No difference.� PMR is open with IBM.
0 - Got the OK to boot clover and basil last night.� It turned into a complete clusterfuck that was my fault entirely.
� - A few weeks back, Slick deployed these VMs and did the identity change and tool install etc.� He's not a UNIX person, but he knows his way around well enough.� After he finished, he asked me to double-check what he'd done and I completely blew him off because I was working on other things.
- The servers rebooted, but came up complaining about VMWare Tools not matching the running kernel.� I think Slick compiled VMWare tools before shutting down, not after restarting.� So when he did the compile, he was still on the old kernel.
- I booted them both back to their old kernel and got networking up, then loaded in a compiler and tried to find the kernel headers or source code on RHN.� It wasn't available and I have no idea why.� Perhaps the kernel was retracted, I don't know.� At any rate, I finally got the source code from kernel.org and got it through our firewalls and into a system where I could move it around.
- Right in the middle of this, a storm blew through the desert and knocked down my business Internet connection but spared my home connect even though they're both in the same house.� grrrr.
- Booted the systems to the new kernel, compiled the tools, unloaded the source & compilers and turned them back over to Becky to get OV up and running.
Ended up taking 1.5 hours instead of 20 minutes.
tada!
Becky is out sick today
Cartman is working from home.
First order of business today was to setup the satellite acct with sudo access throughout nonprod.
- Talked with Slick about the time issue and let him know apollo was 15sec off.
- WICK - Note from Ramstein that nfs mounts to fry were down.� Pulled up the wick docs (because apparently the ops team can't handle this simple task) and found out that it gets nfs mounts from eros, dionysus, poseidon, and atlas.�
- Stapho says basil & clover aren't allowing monitor to do anything.� no remote commands.� Logged into each machine and su to monitor.� Everything seems normal.
- Check logs and note that BOTH servers are logging as thyme (their clone master)
� Did Slick or I reboot these after changing their ID?� Neither of us remember.� Need to reboot them now to make sure.� They've been up 19 days (aug 5).
� Can't bounce, they're PROD already.� Jumpin' jesus on a pogo-stick.
- So I fixed the problem with WICK and sent a note about it.� Of course Pissy Wick Boy had to ask a bunch of pissy technical questions that he won't understand the answer to anyway, but he likes asking them.� Maybe he thinks it makes him seem smarter or something.
- Tequila doesn't have an admin ID to get into virtual center in madison or new york.� fecking feck.� I can't show him how to upgrade the kernels then.� I'm starting to have my doubts about that boy.
- We have to come up with a solution for LDAP and NFS /home shares SOONish.
- Sent Arnold the Pig some info from crow & petrel
- Skipper needs me to give Triscuit some sort of "evidence" that I shut off root access.� Requested clarification.� WTF kind of "evidence" do they want?
- Cheesedog's change got nixed for tonight so I'm not bouncing clover or basil.
Call from Cartman:
Problems with NTP in nonprod, he's locking firewalls and sees some strange calls.�
Investigation / background
- All of nonprod syncs ntp with a vip named 'ntp' in DMZ and 'time' in CORE
- CORE/time vip points to NW for time
- DMZ/ntp vip points to non-prod server skylla on mage
%%
With the dmx migration Slick and I shut down NTP on all vms and instructed them to sync time with their ESX host.� All ESX hosts were pointed to 'ntp' for their time, under the mistaken assumption that the 'ntp' vip also pointed to NW for time.
We also shut down ntp on skylla.� This effectively left all of nonprod without a timesource.� It will drift.
I asked Cartman this:
"If it's a known issue that Virtual Guests don't sync time properly using the NTP protocol, did it ever make sense to use a Virtual Guest as a primary time source?"
He agreed that it does not.
Cartman volunteered to assign NTP duties to our nonprod Cisco switch and to point the ntp VIP to it.
That means all of our Virtual Guests will sync time with their ESX host and the ESX host will sync time with the 'ntp' VIP, which will use the nonprod Cisco switch as its time authority.
%%%
Cartman will get this set up "sometime next week" because he's "busy".
Meanwhile, time on these machines will drift.� I ran through today and punched it down for all of the nonprod boxes and found them all .5 sec off.� Apollo was 15sec off.�
Friday Slick & I cut over the DMX from 2 to 3 on nonprod portal.� We had an issue with Mage kernel panicking on reboot so after much troubleshooting and consideration we decided to do an in-place upgrade from ESX 2.5 to 3.02 and migrate it over to Madison and off of Fairbanks.
This worked pretty well.� We had a couple of outstanding things to do:
- Full Patch load
- New Kernel Load
- Set clock=pit in kernel and shutdown ntpd
migrate DMX
- ntpdate (lockdown time)
- recompile vmware tools & restart networking
- remove gcc & kernel-source
- remount nfs & smb shares
- visudo (add satellite acct)
- chgrp & chmod /bin/su
- ntpdate one more time
- I need to change the password for the satellite account everywhere.
o we can use that account to spray files through the env
- set up at jobs to update systems at 4am, prior to esx/dmx cutover
o change root in prod
Drove in this morning a little late.� I had problems sleeping Sunday night and Monday was a lazy/slow day for me.� I knew my calendar was booked for the DMX work most of the day so I decided to sleep in a little and drive in by 10.
10:15 - The WICK guys opened a HEAT ticket for their ID creation in Dev.� Cartman started to ask me about it and I had to dump him off to Hoppy.� I don't create the IDs in LDAP.� They have to open a snap request and send it to Hoppy.
- Mr. T. needs draft budget information by the end of the day.� Sent him a summary of what I need for 2009, none of it is outside the lines so there won't be any approval issues.
10:35 - We started the DMX migration on-time and picked Okoboji as our first conversion.� System down and Slick & I walked across the street to get coffee.� Sounds like Margie is just about done re-zoning the disk so we're going to try to bring it back up.
Crazy weeking helping A move her stuff, some drama to discuss with Chez about her relationship, and went out to dinner with GoodBoy to celebrate his exam success.
- 12:45 - First update possible this morning.�
- There were� Portal issues this morning related to the log space on the Minotaur cluster.� Apparently it sent out warnings about the situation but� nobody got them due to our "private time" settings on weekends.� (we don't like to be disturbed on weekends, so on Sunday nights, all pages are held between 11pm and 5am.� Even on critical systems.� d'OH!)
-Network issues in the office related to the installation of new carpet.� I drove in this afternoon to do a Satellite presentation to the OPS guys, and because we're interviewing another candidate today.
- Working on an on-boarding document for new folks.
- 1PM - Satellite demo is just about ready.
- Satellite demo went well and I'm going to patch the Dev Portal on Tuesday.� Actually, I'm going to see if Chops wants to do it.
o - Interview went well.� Everyone agrees that this guy is a better pick than DevilBoy so I'm pretty sure Mr. T. is going to extend an offer.� Good for us!� :)
4pm - Going home for the day
Started out Thursday at home but Slick wanted me to drive in for the ESX remediation/redesign meeting.� Mostly as a show of solidarity since I'd already been collaborating on the project and I've seen all the data.
Had lunch with Cartman, Sper-dooge, and Grandpa Simpson.� Went to the Greek Restaurant up in north Scottsdale.
The presentation went fine, everyone agreed to move forward and we only spent 3 hours on it.� I spent the next couple of hours talking with Mr. T and Slick about various items of business.
- Brought the 64 Oracle server home to build at my leisure.� I want to get it done next week if I can, right now its just sitting on the spare chair in my office.
- Mr. T. said that if I _really_ want to move to Oregon, he might be able to persuade our CIO to let me continue working for the company and just commute to Phoenix once a month.�
-- I could buy my own plane or just fly Southwest Airlines, the only airline still in business that makes a profit and has loyal customers.� Maybe its because they treat the customers like they matter.
Good morning.
7:30 - talking with Becky on IM
- Clover & Basil need to be added to the backup rotation.� Opened ticket 469134 to (ch)InkPen and pointed lgto to haydn for backups.
- Becky already talked with Staph-O about monitoring.
8:00 - Emails floating about regarding IM00105055 and the SMTP routing of inbound and outbound mail through our parent company's Domino servers.� This is probably what's killing my POP3 on the Blackbird project and I sent an email to Rob (THE HORACE GUY) about it.
8:30 Not sure if the ENG/OPS meeting is still on, but I'm double booked with this and PD2.� PD2 will be priority.
- No call yet from wick prick regarding the changes made to the DOCU boxes.
8:55 - Need to add a few user accts for people in Dev:
Prince Chili Longhorn: sudo access
Wick Prick : account in dev portal with sudo
9:00 - Things I NEED to get done today
- Switch off password prompt on sudo in non-prod
- verify nsswitch.conf entries are files ldap winbind for passwd & dns files for hosts
**********
9:00 - ENG/OPS meeting.� DC got all hung up on my desire to shut off the password prompt for non-prod.� I really wish that people would refrain from offering opinions on technology that they neither use, nor understand.� kthxbye.
10:15 Rob (THE HORACE GUY) says to use� entddcmfs01 instead of ohcollnp039 for PFF so I forwarded that out to Prince Chili Longhorn and Tigger.
10:30 - Finally joined the team meeting, already in progress.
�12:55 : Non-prod: verified nsswitch.conf orders and set PermitRoot to NO in the sshd_config file, restarted sshd.� For NON Prod, all direct root access is officially gone.
- The wick Pricks are still testing my script changes.� They had one weird issue which Chops couldn't find a reason for and they can't reproduce it, so we're going to assume all is OK.�
- Just for clarification, the changes I made to their scripts make the WICK DOCU environment function the way it was originally designed and documented.� I have remediated the jackassery done by whomever thought they were clever and re-wrote them to do something they should never have done to begin with.
- Friday Night Change control - Must fix prod scripts for WICK ::before:: altering the the sshd_config or changing the root password in Prod.
1:45 - Becky needs me to check the OV environment to make sure the mail relay hosts are pointing to the right place.� apricot, iris, basil, and clover are the kids I checked first - apricot has 'mailhost' instead of 'mail', but they resolve the same so I'm okay with leaving them as-is.
August is always the worst month for me.
I've been sick AND having to spend more time IN the office, which doesn't give me time to make web notes.� They all go in my hand-held journal.�
o Interviewed DevilBoy for a Windows Engineering position.� I like him and think he'd make a good addition to the team.� We have 1 more interview to do (tomorrow), but so far we already like candidate 1.�
o WICK exploded on Monday.� Imploded?� The Dev & QA environment Content Push scripts failed because someone changed them.
�- Documentation showed that the DOCU server had NFS mounted shares from Oceanus and PAN.� That means when OH servers write out to DOCU, the data is immediately available on Pan and Oceanus.� Instead, some JACKHOLE set up rsync to copy the files rsync over ssh into Pan & Oceanus.� So when I changed the root password on those servers...BAM.� All their scripts stopped working.
- Thanks Lloyd.� Your incompetence has bitten me right in the ass yet again.
- Spent most of the day analyzing the issues and playing phone tag with the WICK pricks.� 5:20PM the last script edit is done and I sent out notification to everyone back east to try it first thing in the AM.� If my fixes continue to work without issue, then Chops and I are going to run through QA and let the pricks test it.
- Script changes will be done in PROD on Friday before I change the root password.
My day is done.� I had planned to go into the office on Wednesday for the interview but considering how badly my body reacted this morning, I think I may stay home and do it via conference.
-- I interviewed two new candidates for an assistant, one on Thursday another on Friday.� There wasn't a lot of actual work being done, just mostly meetings and interviews.� I settled on the first guy, his name is Jeff.� With any luck, Mr. T. is making him the offer about now.
�- HAPPY FUCKING AUGUST.� (August is historically the worst month for me personally)
09:05 - Meeting with Ditty, Squirrelface, Hoppy, and Hoops about the outstanding portal audit info.� Squirrelface is tossing out items that failed audit and I have to address them.� One of the items is that "su permissions are 755 and should be 500".�
it's idiotic.
- 09:06 - Call Brett.
- Squirrelface doesn't know what any of these audit items mean in a real-world application.� The items on the list are just silly to me.� Default umask is 022 and they want it to be 027.� Whooptie-shit.
10:05 - Reboot of my workstation.� Trying to connect to VDI kicks me out with a 503 error.� Stephanie @3600 tells me it's because I'm using the "super secret engineering proxy" @202. and I need to switch it to confserver.� Poof, I'm in.
10:15 - Mr. T reminds me AGAIN to log my time.� Bah!� Why do I always blow this off? :)
10:30 - Added Brett to wasadmin group on nymph to see if he can get the rsync CMS stuff working.
10:35 Mr. T. said Jeff doesn't want the job.� Too stressful.� hah, fuck.� What a puss.
�1:10 - Had rice for lunch.� yay. � - Interviewing TC as a formality.� He's an internal guy who wants to transfer to this open REQ, but he's no where near qualified for the job as far as I know.� I have a ton of questions qued up that I hope he can answer.
00 Spoke with my friend Mitch.� He needs a job reference.� Gave him the OK to disclose my contact info.
3:30 - Rejected TC due to his lack of UNIX exp.
- Gave Mitch a GLOWING reference.
- Grabbed stats for Gunslinger on Thyme136
- Good Oscar called me about the K&K project.� I asked her to schedule a meeting with Prince Chili Longhorn, myself, and Becky.� She's going to make it Wednesday @1.
- Something is wrong with LDAP on Oceanus.� Brett & I get in but can't sudo.� It's pulling the info from the e-dir server so I don't know what the deal is.� Asked Hoppy to check into it.
:: recap ::
-- Set up interviews for Friday morning.� Two guys to interview for the WICK opening.� Both candidates look somewhat over qualified and will probably not enjoy this job.� Guess we'll have to find out.
-- New Project start: K/K-A� --� house with PFF -- Prince Chili Longhorn and I will be building this.� Looks pretty simple from the documentation I went through so far.
-- Mac couldn't get into skylla.� Explained to everyone that this is why I wanted to do the PM servers at the same time as the rest of the PA and PP systems.
-- Discussion with Cartman regarding our LDAP problems.� He's going to schedule meetings so we can get a clearly defined issue and then we're going to root out the solution.
-- Slick is in training this week.� I think he's going to come in for the interviews Friday.
-- Carrot Cake wants to discuss the WICK systems because she's freaking out over the RSYNC discussion.� Everyone is freaking over it and she wants me to look into it.� I pull up the visio diagram and after a few minutes tell her that she's okay.� The diagram shows rsync happening bewtween leela and some other box up in Ohio, but nothing that touches the Greek systems.� The only thing touching the Greek boxes is HTTP and NFS.� They're going to be fine.
- Took the time to explain to Carrot that we're only 7 people trying to keep a $2B datacenter running.� If they don't get us more hands, people are going to have to wait longer for solutions.� (And that is true.� We have 1 UNIX engineer (Me), 2 support staff augmented by 1 contractor, and we have 1ESX engineer who has 2 support staff.)
weird.� I dunno what happened to the entry I was making.
- I have a schedule lock today for PFF from 9am to 3pm.� I'm only on "standby" for the code migration as the regular support folks are going to work with SS and learn how it's done.� I have to be ready for problems but I'm confident it'll go smoothly.
- This gives me time to work on other projects.
0- A bunch of CritSit notices went out for the Windows systems.
12:10 - Squirrelface wants to know why his auditing program can't log into Pan as root.� I'm just _floored_ by this.� He's an idiot.� This man is our Risk Management specialist.� He's the shop "security guy".� He's been involved in my project to shut off root for the last 9 months.� WTF is he thinking?
- He asked me how to set someone up in the group 0.� !!!
- This is what I have to deal with.� THIS.� Sometimes it makes me want to put a gun in my mouth and paint the walls with my brain.�
1:00 - Pam/LDAP meeting starts 10 minutes late.� I'm already pissed because of Squirrelface, and now everyone is late to the meeting.
1:10 - Fuck this.� They can call me.� I'm hanging up.
1:16 - Bastards.� Meeting start.
- If the people who audit our systems are going to ding us for allowing direct root login, then they shouldn't send us an auditing tool that can't run unless it's allowed to have direct root login.� I tell them to go fuck themselves and come back to me when they think their cunning plan all the way through.
2:30 - more ICD changes need to be done to Pan.� I realize my entire rant there makes me seem angry today, but I'm not.� I was just annoyed by that situation.� :)
9:50 - Timesheet in for last week.� 58.
- Pooh Bear (Whom I'm told has another nickname in the shop.� People call him Burt) asked me to fix perms on /orabackup on G/T.� Done.
-� Becky said she opened a ticket regarding slowdowns in Portal Prod
- CountryBoy hits me with this little gem:
CountryBoy: So, on Juno ... /home/XXXX directory ... permissions got hosed.� I'm trying to run a scripts and it's erroring out trying to delete some files in that directory.� Says 'read only file system'.
CountryBoy: Can you help, it's the content update script that's bombing.� Got something that needs to go to prod.
CoralineBlue: � Ok.� I'm looking.
CountryBoy: Thank You so much CountryBoy: Can't chmod the files, even logged in as root.� One of the files is XXXXX.XXXX if that helps. CoralineBlue: �� ok, sec CoralineBlue: ��� oh boy
-- Looking @ Juno I see what he's talking about.� Can't read/write from that directory.� But that directory is on /media/hd2, which is a separate partition on a HUGE sanlun.� I umounted it and it gave errors.� I try to remount it:
[root@juno root]# mount -a mount: /dev/sdb1 is not a valid block device [root@juno root]# fdisk /dev/sdb Unable to read /dev/sdb [root@juno root]# fdisk /dev/sdb1 Unable to open /dev/sdb1 [root@juno root]# fsck.ext3 /dev/sdb1 e2fsck 1.32 (09-Nov-2002) fsck.ext3: No such device or address while trying to open /dev/sdb1 Possibly non-existent or swap device? [root@juno root]# fsck.ext3 /dev/sdb e2fsck 1.32 (09-Nov-2002) fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open /dev/sdb Could this be a zero-length partition? [root@juno root]#
Called Slick to look at the storage on ESX host druid.� I asked him to detain and quarrantine Grandpa Simpson just because I KNOW he had something to do with this.� lol.� :)
- 11:30 Someone went into Navisphere and removed the disk from Juno...probably this morning.� I was using the system till 2am on Saturday, so I know it was recent.� Margie put everything back and I bounced Juno and its happy.
- Hoops asked me to dork with the nsswitch.conf settings on elf to see if we can get around this ldap auth timeout issue for sitescope.
Add comment
