収束ゾーンでのライフ: Life in the Convergence Zone

Today Anna and I upgraded all the servers in the servercore with MST3K in the background and had some pizza made by Paul and some pie made by a grocery store.

And it was a really nice day.

I am so pleased that (with the help of the Mastodon discord on that server) we got all the server OSes upgraded and back up and running and cleaned up some stuff while we were at it.

Just being able to… get some stuff done without it being a goddamn nightmare or a huge fight or anything. Just mostly sitting around and chatting some while upgrading stuff and getting things that broke working again without too much of a fuss.

could use days like this more often, not gonna lie

how was your day?

Posted via Solarbird{y|z|yz}, Collected.

Massive windstorm this evening in Cascadia has taken out our power; we lasted, honestly, longer than I thought we would. But we are definitely now out. This should federate out to most of our followers, though, since we have like an hour of battery backup that we’re chewing through right now..

We’ll come back when power does. If you’re local, well, good luck out there.

Posted via Solarbird{y|z|yz}, Collected.

I haven’t seen this mentioned here but if you haven’t updated Windows this week, DO THAT NOW

Like seriously, right now.

no

right now

Posted via Solarbird{y|z|yz}, Collected.

2024/2/14 10:13am: I’m so glad that worked. We’re back. And we also have a security update to apply. I can handle that, as long as we are, as they say, _back online_.

2024/2/14 4:09am: It’s not a great overnight here at the mighty mighty murkworks. Our mastodon server, mastodon.murkworks.net, has been taken down by FUCKING SanDisk; the drive went from “Everything’s fine!” to “lol fucked” in 0.0 seconds, panicked the kernel, marked itself “24 hours good luck” and closed off the top of the well.

The good news is the drive is responding to queries and I’m currently ripping the important partitions. If this doesn’t work, yes, of course we have backups, but that’s a lot more hands-on and I can’t sleep through it like I can this. If things go very, very well, we’ll be back up… maybe 10am Cascadian/Pacific time Wednesday morning? Probably closer to 11.

I do see replies to this post and can reply to those replies. Ironically, the only people who can’t see this are the people who need to see it most, our users. Lol.

This post will be updated as events warrant.

— @moira

Posted via Solarbird{y|z|yz}, Collected.

BEST THING: when your networked ancient but non-DRM laserprinter has been crashing on certain PDFs for months so you think there must be something new in PDFs and maybe it’s replacement time BUT

you discover your printer has room for RAM expansion so you check and it’s at minimum configuration so maybe it’s just running out of RAM so you buy some old compatible RAM off eBay BUT

before you install it you try to find some old PDFs that crashed the printer so you can know if new RAM actually fixed anything and you try printing them again before installing the RAM because you’re smart like that AND

…none of them crash it anymore.

goddammit printers

why

Posted via Solarbird{y|z|yz}, Collected.

I

finally

got iOS to let go of an expired certificate that it kept insisting upon using in place of the actual cert on the server

and

finally got both the mail and mail-relay servers to use the automatically-updating certificates

and

got the auto-updating certificates to maintain usable permissions so that all of the above could actually happen.

goddamn that was a pain in the arse.

From Twitter, as usual for these rants where I'm trying to get a company's attention.

So today's @ComcastBusiness adventure is they want to charge us over a thousand dollars (>$1000) to disconnect service.

That'd be the service they kept breaking on their end, taking us offline 2-6 days at a time, four times between October and January.

https://solarbird.dreamwidth.org/tag/murknet

The link points to writeups of their failures.

I must note that we did sign a term contract at a time when we had no other options, but still: we sign up, @ComcastBusiness starts taking us down over and over again for days each time.

Without compensation, if that matters.

Now to me, that sounds like breach of contract, failure to provide service.

I mean, sure, service is there the _majority_ of the time, but @ComcastBusiness really did make that "nine fives of service" joke into reality, and that's unacceptable business internet service.

The cherry on top of all the other cherries on top - ignoring us, closing trouble tickets, telling the woman who ran a division-level mail server room at microsoft to "talk to [my] local IT about this," and so on - was trying to charge us $100 to fix their mistake the last time.

We _were_ able to make @ComcastBusiness reverse the $100 charge for fixing the latest mistake _they made on their end_, but now that we're finally set up on another provider, they want $1000 to go away.

We're considering our options.

Now, I'm not saying I'm asking for a legal opinion, but I am wondering whether @ComcastBusiness's repeated failure to provide service on this scale without compensation does in fact breach contract sufficiently to void the contract.

@KathrynTewson for no reason in particular ^_^

I also know we're not the only ones @comcastbusiness have treated like this. I've talked to others who just gave up and walked away. Maybe we'll end up doing that. This has been a several-month nightmare, and I'm an individual, not a corporate monstrosity, with other work to do.

And @ComcastBusiness pays people at scale to do this sort of customer-screwing, it's their jobs.

Mind you, we were even willing to keep the associated phone service. We wanted to move it consumer since it's not actually a business line, but we were willing to keep it, because basic POTS? That, apparently, @ComcastBusiness can manage.

But internet with fixed IP? Hoo boy no.

And that's when @ComcastBusiness said "Pay us $1000 to go away."

What an absolute trainwreck of a company.

Anyway, like I said: we're considering our options.

But I know this much:

Friends don't let friends do @ComcastBusiness.

eta later:

I just thought of something vaguely amazing about this $1000 BS.

We said we'd keep phone service, albeit moved to residential. (It's not really a business line.) That works out to $1500 in their pocket over the contract.

$1500>$1000.

@ComcastBusiness can't even chisel right.

This is long past being funny.

Dear @ComcastBusiness, regarding your latest blowing off our 3 days and counting of complete lack of IPv4 connectivity and your several closed tickets, the latest of which being CR020322088:

"I have verified that all of your Comcast internet services are working correctly."

Your gateway is moving ZERO IPv4 PACKETS. NONE. IPv6 moves fine. IPv4 does not.

I have told you this, repeatedly.

"I am able to login to your Comcast gateway and reach websites from within your gateway."

Yes, I know.

Do you know, @comcastbusiness, who told you that IPv4 could communicate with the gateway?

I DID.

I. TOLD. YOU.

The problem is IPv4 can't _cross_ the modem from the LAN.

"I am getting solid pings/signal from your Gateway Static IP and all signal levels are in the green."

Yes. I know. How do I know? I TOLD YOU ABOUT IT.

When I could log into the gateway.

Which, from the LAN side, I often can't. It refuses all logins.

I told you that, too.

"There is nothing I am finding that would cause the intermittency you are experiencing."

What we are experiencing, @ComcastBusiness, is COMPLETE FAILURE TO PASS IPv4 AT ALL TIMES.

Being able to pass IPv6, while cute, is not "intermittency." We have been down for THREE DAYS.

We have been down, @ComcastBusiness, for THREE DAYS, with ZERO PROGRESS ON YOUR SIDE.

This is the FOURTH MAJOR OUTAGE SINCE OCTOBER and you are STILL not LISTENING TO US.

"The best way to test your services would be to directly connect a single laptop or PC to the back of the Comcast modem... and run a speed test at http://speedtest.xfinity.com."

Hard to run an IPv4 speed test when you pass zero (0) IPv4 packets, don't you think?

"If you direct connect and are getting the correct speeds (100MBPS DOWNLOAD/15 MBPS UPLOAD) this means the issue could lie somewhere in your networking equipment or we have too many devices connected."

OUR SPEED IS ZERO.

YOU ARE PASSING NO IPv4 PACKETS FROM THE LAN SIDE.

NONE.

What's it take, @ComcastBusiness?

In the last three days you've told us our modem is broken but you won't send us a new one, that our modem is fine and that it's our fault (spoiler: it's not) and that we need to do speed test on a modem which passes NO PACKETS from the LAN.

And - and - @ComcastBusiness you have to understand - your side has managed to diagnose NONE OF THIS.

I told _you_ the modem can pass IPv4 from _itself_ to the WAN. I'm fully aware the modem can pass IPv4 from itself to the WAN!

The problem is it can't move them from the LAN.

We got the same runaround the previous 3 major outages since October too, btw, @ComcastBusiness - but this one's even more insulting because it is a _complete_ outage. The last time, we had _some_ service.

Now the only service we have is IPv6 and attacks on my blood pressure.

At this point, @ComcastBusiness, we're three (3) days into a complete outage and you're back to telling us nothing is wrong because some intern managed to get the modem to send an IPv4 ping.

Which I TOLD YOU IT COULD DO.

Here's me telling you, @ComcastBusiness, on Friday, that the modem could pass IPv6 traffic but not IPv4 from WAN to LAN, and that the modem itself could pass IPv4 traffic. Remember this? You confirmed it was added to the ticket!

Of course, you closed that ticket too.

"I would recommend that you start with basic troubleshooting, and restart all of your equipment or contact your IT professional."

I RAN A DIVISION-LEVEL MAIL SERVER ROOM AT THE MICROSOFT CORPORATION.

I AM THE LOCAL IT PROFESSIONAL.

YOU KNOW THIS. My god.

(It was the smallest microsoft division at the time, but nonetheless, it's still true.)

"Alternately we can schedule a technician visit to verify that your services are working properly but if it's beyond the modem the technician can charge a $99.95 service fee due to not being a Comcast issue."

@comcastbusiness why you gotta hurt me like this?

You told us last night @comcastbusiness that you could see the modem was malfunctioning and needed to be replaced.

You also told us you wouldn't send a technician and also wouldn't send us a replacement modem.

Is this @AGOWA time or what?

Are you ever going to move IPv4 packets again?

I need to know, @comcastbusiness. This is our 4th major outage since Oct, all of which you blamed on us but were your fault, & you won't even acknowledge a problem. We are 100% down for 3 full days and counting.

Is it @AGOWA time?

Because at this point, @ComcastBusiness, I kind of have to act on the idea that you're never moving another IPv4 packet again and we're just going to be down for... at least another week. Probably two.

Friends don't let friends do @ComcastBusiness.

Hey, @ZiplyFiber - are your sales offices open on Sundays? Because I got some answers from your business tech people last night without even having a business account and they were the right answers.

The photos are gone because fucking Comcast Business keeps taking us offline and can't get us online again for days every time they do it.

The photos will be back when Comcast Business takes its boot off our connectivity's neck again.

Yeah. I tagged 'em. At this point, why the hell not?

SICK OF YOUR SHIT ROLL CALL:

1. "This isn't on our side, you should talk to your local IT about this."

A: It's been on your side EVERY TIME. Eventually, Tier 2 fixes it.

B: I AM THE LOCAL IT. I ran the WGA division server room at Microsoft. I HAVE SOME IDEA WHAT I'M DOING.

2. "Have you power-cycled your modem?"

Classic.

3. "Are you sure you have the right password?"

You reset it to factory default. Three times. It won't start magically working.

4. "Windows should be restarted."

My fleet of Linux servers glares at you angrily.

5. "I'll re-provision the modem. Wait five minutes and try again."

That didn't work the last four times, why is it going to work now? Oh look, it didn't.

6. "Someone will call you Monday."

[No call]

"Tuesday"

[Calls wrong number]

"Wednesday"

[Tier 2 shows up and fixes it]

(If we're lucky. I think the quickest we've been back up was two and a half days.)

7. Me: "[X] in Tier 2 told us you need to do exactly this, and can do that."

Tier 1: "We can't do that."

[tries again]

Tier 1: "All done!"

[nothing actually done]

8. [support wakes up from idle after hours of silence and apparent inactivity] “Is it fixed yet?”

No. No, as you have apparently done nothing, it is not fixed yet. The problem will not magically go away. It. Is. Not. Fixed. Yet.

Honestly, and I mean this in all sincerity, it has felt _many times_ like we've been being trolled.

Tier 2 is generally good to great and totally know what they're doing. Nice people.

Tier 1 - by phone and online - was not always a nightmare. I remember this. But it is one now.