9:00pm It is back online now and working normally
6:30pm We are having problems with the Henry and Rivington hubs. This is affecting downtown Manhattan connections
The new multipath configuration is now complete in Brooklyn and many are reporting fast speeds tonight.
There were further adjustments to the Brooklyn network today as we standardise various hub nodes. Hopefully when we add back the Soft Suplus gateway we will be back to normal bandwidth tomorrow or the next day.
2:00pm We have fixed some of the bandwidth problems. There are still some isolated slowdowns. If you are having problems please fill in a support ticket
5:00pm Bandwidth is low in Brooklyn until we fix our Soft Surplus gateway, hopefully later tonight or tomorrow.
4:30pm We have bypassed the problem and most nodes (96%) are now back online. We are still working on it.
1:40pm Our Saratoga hub is down causing outages in Brooklyn. We are looking into it at the moment.
3:30pm Upgrade completed successfully at Saratoga, and point-to-point connection established to SN4
11:30am We are performing an upgrade at Saratoga as part of an on-going effort to increase speeds. This will only affect people connected to that Brooklyn hub. The hub will be down for 30 minutes around noon and occasional ups and downs in the early afternoon.
Unusually heavy rain is interrupting connections downtown and in Greenpoint this afternoon. We have a large upgrade planned during August and September that will fix these rain problems
12:59am Bandwidth has returned to good speeds in much of Brooklyn due to improved configuration of the SN3 VPN server. This also fixes much of the Saratoga problem.
Heavy rain has been interrupting our Henry and Rivington connections downtown. This affects most nodes that are not directly connected to Supernode 1. We are planning a fix for this but it will take a few weeks to implement.
Saratoga Hub is still having bandwidth issues. We are about halfway through fixing this. There is now a fast connection from Soft Surplus to Saratoga. Next we have to upgrade the main router and also add a second West-facing sector antenna. This should all be completed over the next 4 weeks.
3:45pm Fixed the problem (actually a bad port). Connection in Brooklyn should be faster now than it has been the past week.
2:40pm Replacing a bad cable. Brooklyn nodes may be down for 20 minutes or so.
3:52pm Connection is back up. All nodes online
3:24pm Henry St hub is down affecting downtown Manhattan. We expect it to be back up within the hour
8:30pm Rivington is back online.
8:00pm The Rivington hub has disconnected causing outages downtown and at node 2090 in Greenpoint
9:38am The Saratoga hub gateway has been switched from Ehud to a 600 Mbps connection at Soft Surplus. This will improve bandwidth and latency for much of Brooklyn.
7:25pm Back to normal
6:20pm Downtown Manhattan is experiencing outages at the major hubs. We are looking into this at the moment.
11:50pm We’ve just fixed a problem at Soft Surplus that was causing slow speeds in Brooklyn since this afternoon.
2:45pm We are replacing a cable at Henry hub. This means there will be 3 minutes downtime while the router reboots. After the cable is replaced speeds from downtown hubs should increase significantly.
3:45pm: Fixed the problem, just a reboot needed. Back to normal now.
2:10pm: We have a problem at Henry hub affecting Rivington, Ave A, 7th St and Ave D hubs downtown Manhattan. Looking into this now. We should have a fix by 4:30pm.
12:10am: There was an 8 minute downtime due to an internet exchange problem. Back to normal now.
8:00pm: Things should be connected again and bandwidth should improve over the next few days.
12:00pm: We are having problems with our Soft Surplus hub node (1417) in Brooklyn since midnight. We are looking into this now. This is affecting the 9 connected nodes.
4:30pm: Normal service resumed downtown. This was a problem with integrating our new mesh configuration with some older installs.
4:25pm: Our Henry St hub is having problems and this is affecting many of our downtown nodes. We are working on fixing this now.
11:00am: We had a problem with our internet exchange for the past two hours. It appears to be back to normal now.
Five nodes connected to Ave A hub were down around 2pm today due to a configuration issue. They are back up as of 6:20pm
1pm: We rebooted the Rivington sector and it is working normally again
11am: One of the Rivington Sectors is having problems disconnecting 100Ava and about 5 connected nodes in the LES and East Village. We are working on this at the moment
Brooklyn is experiencing slow speeds especially around 8 to 9pm (“Netflix time”). We are working on fixing this.
A few people are having problems with LinkNYC kiosk using our new SXTsq VPN config. We are trying to solve this problem, but it might be better to switch back to the non-vpn config for now.
There will be a couple of short interuptions between 2pm and 3pm today as a cable is changed at Supernode 1
1:43pm maintenance completed
Due to maintenance at one of our main Brooklyn hubs there may be some temporary interruptions to between 12:30pm and 1pm today
6:00pm Power restored. All connections should be good now
5:45pm Electricians have arrived at the Saratoga building and will be working on this tonight until fixed.
9:24am There is a power outage at our Saratoga hub. This is causing problems in a large part of Brooklyn. We are sending someone out now to look at this.
DE-CIX New York, our internet exchange point is giving us new IPv4 addresses. This will cause a small amount of downtime as we switch addresses over the next two days.
Change period starts: on 2019-01-28 at 10:00 EST (15:00 UTC) End of change period: on 2019-01-30 at 07:00 EST (12:00 UTC)
Finished connecting Henry to SN1 using 60GHz gear, this upgrades Rivington connection to about 800Mbps.
Antenna alignment today at Ehud to fix ongoing bandwidth problem at Saratoga in Brooklyn. Bandwidth should improve in the afternoon after a short downtime around noon.
Rivington hub is still in process of upgrades. This is affecting 4 active nodes in the Lower East Side. We are planning to get full bandwidth there next week by connecting to Henry hub and SN1 using new gigabit 60Ghz gear.
Fixed node 1417 connection at 5:45pm
A storm took down our node 1417 and is affecting 3 nodes in the East Williamsburg area. We hope to have this fixed tomorrow.
Problem fixed! All ok.
A failed upgrade caused downtime at the major hub node in Brooklyn this morning. It was fixed at 4:15pm today.
You can also get updates in the #1340-alerts channel on Slack.
Recently our Supernode 2 had to be taken down for roof repair which forced us to make major changes to the area’s network. We are still working on improving the new connections. The good news is many new hub nodes are coming online, so there will be more choices to connect to in the coming weeks.