10:00am We are currently working on a problem at our Henry Street Hub. This is affecting some connections on the Lower East Side
Heavy rain has been causing problems in Brooklyn. We are sorry about this!
Longer term we are upgrading equipment and adding leased fiber to eliminate this problem.
At 12:40am on 8/11/2023 we experienced a power supply failure at our SN3 datacenter. Due to limited power availability, the antenna for Sunset Park Hub (Node 231) is currently offline. This outage affects all nodes connected via Sunset Park Hub. We are working to get a volunteer on site to restore power and expect to have to situation resolved before the end of the day.
8:00pm backup connection is now functioning well. If you are having any problems at all, please go to nycmesh.net/support so we can quickly fix it!
5:00pm Our SN3 Supernode in Brooklyn is currently having problems with our transit connection due to ConEd work. We have almost resolved this problem with backup connections. ConEd work is due to be completed by 8pm tomorrow (Saturday)
9:00am Ave A now has backup power so it is functioning normally.
5:00pm Our Ave A hub has a power outage. We are waiting for the building to fix this, hopefully tonight. This affects some people in the East Village above 7th Street
5:40pm We just fixed a problem with a slow connection affecting some people on the Lower East Side. You should be back to normal speeds now!
3:15 am (started) the exit via sn1 is not working which seems to have an effect on the DNS server as well. If you are connected to SN1, henry or 1971 (twobridges), exiting via SN1, your service is out for the time being. For the others it should be working with maybe some DNS issues. You can adjust your DNS address if needed or post on slack #support and we can check for you.
7:15 am Traffic has been rerouted. Service should be back.
12:00pm We have repaired a fiber that goes to floors 2 through 7 of 410 Grand Street. Sorry about the short outage. Everything should be good now!
10:10 PH hub is experiencing a power outage. We are working to fix it. Stay tunned.
12:25 PH is back online. You should have your connection restored.
2:30pm We finished the major upgrade at 111 8th Ave, resulting in increased speed and reliability across our network!
1:00pm We are planning an upgrade at a data center this afternoon. This will result in higher bandwidth for a large part of the Mesh.
2:00pm There was a one hour problem that affected downtown Manhattan bandwidth. Sorry about that. It is now fixed.
We are still upgrading the network. If you have any problem at all please fill in a support ticket immediately. This will help us a lot!
1:30pm fixed a problem at SN1 which was resulting in slow download speeds in Manhattan
There are planned hardware upgrades and maintenance this week. We will try to do this early in the morning to minimize impact. There may also be periods of lower bandwidth. Today there was a short outage while we replaced some equipment at Supernode 3. Sorry about that!
Due to a hardware problem with VPN, bandwidth is very limited for some locations today. We are working on this!
12:35am There was a 5 minutes outage at Supernode1. It is working fine now
9:50pm to 1am There was a problem at Supernode3 in Brooklyn affecting bandwidth in that area. We have resolved this.
10am to 12:30 Our Ave A hub was down and is running again after a reboot
12:00 noon Grand Street is switching back to the fiber link today after successfully running on the backup link for the last day. The fiber outage was due to a break in the fiber in the street in midtown, which our fiber provider fixed overnight. There may be a minute or two of downtime as we get the link back up again. This should only affect Lower East Side buildings.
10:00am Fixed an alignment problem with our 1971 hub “Two Bridges”. This should improve bandwidth for the 7 buildings connected
1:00am Connection was restored to 410 Grand Street.
9:30am We have an outage at 410 Grand Street due to construction cutting the fiber in ground. We are working on a fix for this now.
7:00pm Prospect Heights should be functioning as normal now
6:00am One antenna is down on our PH hub affecting people in the Prospect Heights area. We are working on replacing the antenna
10:00am We are upgrading our config at our 111 8th POP (data center). Speeds should be significantly faster after this upgrade is completed hopefully later today.
9:35pm The problem is fixed. Speeds should be returning to normal now
6:00pm The network is currently experiencing slow speeds. We are working on it
303 Vernon light pole access points are back online
1pm: All 460 Grand Street should be back online after an issue affecting the top 12 floors
1:30pm Our Brownsville hub is back online after what turned out to be an antenna alignment problem. Sorry about that!
We have a planned upgrade to some hubs in Brooklyn today near Bergen, Revere and St Marks. There may be a short downtime around 12:30pm
11:45am-12:30pm We had a power outage at 410 Grand Street affecting Lower East Side connections. This has been fixed.
12:00pm-12:30pm We had a problem with an antenna that affected Brooklyn for 30 minutes. This has been resolved.
11:30am-12:00 We had a 30 minute routing problem that has been resolved. Sorry about this!
8:00am-8:40am 7th Street has been rebooted to fix a stability problem. The connection should be ok now. This affected Alphabet City connections
1:45pm Sorry for the 20 minutes outage! All back to normal now.
11:30am to 11:50 We had a brief outage that is now fixed. Sorry about that!
The main link to Grand is operating again, which should improve speeds in the Lower East Side
We are currently running on a backup link to Grand Street which may cause slower speeds on the Lower East Side
14:30 Rivington issue fixed (power outage).
13:00 Our Rivington hub is down. This affects LES and Alphabet City and parts of the East Village We are working on a fix.
Our Vernon hub had power issues from 9am until around noon. This is fixed now.
17:20 Power is restored. The hub is back up and issue solved.
15:30 Vernon hub (Brooklyn) is experincing a power outage. We are working on it.
5:05pm Supernode 1 is back to normal operation
4:30pm Supernode 1 is experiencing problems affecting large parts of the Mesh. We are looking into this now.
2:20am There is an issue at the Parallel hub (Greenpoint) which is affecting connections in the area going through that hub. We are working on it. https://www.nycmesh.net/map/nodes/1167-1084-115
11:23pm Simona’s hub is back online
4pm Simona’s hub is currently down, affecting the 7 members connected. This is due to a fiber problem out of our control. This will possibly be fixed around midnight tonight.
2:40pm We are doing planned maintenance in Brooklyn affecting the Vernon and Saratoga hubs. Hopefully this will be finished by 4pm
2:10pm Power restored at Rivington
10:00am Our Rivington hub has lost power. This affects LES and Alphabet City. We are working on restoring power.
Vernon hub was down for approximately one hour due to losing power. Service was restored at 10:05AM.
11:21am Ave A is back online. Reset the breaker
Avenue A hub has lost power, affecting people in the East Village. We are working on this
14:40 power situation has been restored. All devices are back online.
10:40 Experiencing power outage at Saratoga on one of the two power feed, some devices are offline.
Midnight: Speeds have been restored.
7:00pm The network is very slow at the moment. We are working on it
10:20pm connection restored.
10:08pm We are still dealing with the 15 minute outage from earlier. Our network is currently down for ten minutes while while we reconfigure. Sorry about that!
6:47 We have re-routed traffic and connection has been restored. Speeds might be lower than normal while we are working on this
6:30pm We have a problem at SN1 which is affecting downtown Manhattan. We are working on resolving this.
9pm Most websites are available again. We are working on a complete fix.
7:00pm Some websites are currently blocked in Brooklyn by a content delivery network. We have contacted them and are also working on a fix for this.
11:04 1340-Saratoga is back online. More on slack in #hub-saratoga-1340. It was a power issue.
9:40 1340-Saratoga hub is experiencing an issue, probably a power outage. We are looking into it. It is affecting Brooklyn, Bushwick, Bed-Stuy, and Ridgewood
22:00 664-Cypress hub is back up and working normally now.
12:40 Last night at aound 22:00 664-Cypress hub went down. We are working to restore it as soon as possible. You can follow progress on slack in the #support channel, or check here regularly.
4:00pm Our Rivington hub was having problems with a new antenna for an hour or two. We’ve switched back to a working antenna and it should be ok now. This was affecting Lower East Side connections.
6:00pm The bandwidth dropped for Saratoga and SN1 earlier this afternoon. We have fixed most of the problems as of 6pm and you should have your usual speeds now. The cause was some interference today in the city (a “DFS event”) that caused some sectors to switch channels. If you are having any further problems please fill in a support ticket
5:00pm The switch connection was repaired and functioning normally.
2:30pm We have a problem with a switch at Saratoga. This is affecting some people connected to that hub. We are working on fixing it now.
3:20pm The Two Bridges hub is working normally now
12:00pm Our 1971 “Two-Bridges” hub in Chinatown is having problems. This is affecting some people in the area. We are working on this at the moment.
Major upgrade to Saratoga completed. This should improve speeds for much of Brooklyn. Also Grand St was upgraded to a gigabit connection.
20:54 Tomorrow Feb. 6th starting at 12:30pm, we will be performing network upgrades at our Saratoga hub in Brooklyn. Downtime should be minimal – 1 to 2 minutes – but could last up to a half an hour.
20:54 Downtown Manhattan should be back :-) Snow seems to have been the culprit. If you have any further problems please fill in a support ticket
4:00pm Downtown Manhattan is currently having problems possibly related to the weather. We are looking into it now.
3:10pm We’ve finished recovering configurations at SN1.
12:20pm The “ev” sector is now working as before. This has fixed most of the problems.
4:00am SN1 has configuration problems that is causing dropped connections in downtown Manhattan. We are working on it now.
10:30pm fixed the problem by remounting the antennas at 3213
4:00pm still working on fixing the problem at 3213
1:30pm We are currently trying to fix a small outage in Brooklyn affecting nodes 3213, 6385, 1796, 5947, 6012, 6066, 5366, 3360 and 3625.
We had slow speeds this morning in parts of Brooklyn caused by a routing problem at Saratoga. This was fixed around 11:30am
22:45 PH is back online. 22:35 There is a problem with our PH hub in Brooklyn. We are looking into it now.
6:30pm Our Saratoga hub experienced a slowdown for an hour. If you are still having any problems please fill in the support form
8:45am PH is back online. The network is running well.
20:30 There is a problem with our PH hub in Brooklyn. We hope to have it fixed by the morning.
21:00 PH is back online. The network is running well. If you have any problems please fill in the support form
20:15 There is a problem with our PH hub in Brooklyn. We are looking into it now.
The server at SN3 was reconfigured and speeds are now much improved
Saratoga North and West sectors are at a slower speed than usual. This is a problem with a server that will be fixed on Sunday 27
17:55 Rivington North-East sector will be down for maintenance from 5:55pm to 6:10pm
18:20 Rivington NE sector should be much faster now than it has been
12:15pm Fixed. This was an interference problem. We switched frequencies and it is fine now.
11:00am Problem again with Sunset Park sector antenna. We are working on it.
11:15am Fixed the Sunset Park problem. (IP address allocation issue)
9:00am The south sector antenna at SN3 is not working. This is affecting Sunset Park area. We are on the way to fix this now.
1:20pm Fixed. It was a routing config problem through 1971 hub
10:50am We have a problem with the Henry hub affecting downtown connections. We are working on this now.
2:46pm: Power restored at the hub. All good.
1pm: Our major hub in Prospect Heights heights is having a power problem. A member is on the way to fix it now.
4:00pm The fiber has been repaired and Brooklyn speeds are back to normal
9:00pm The uplink fiber for hub node 1417 was cut this afternoon, likely by a truck, causing slowness in northern Brooklyn. A technician will come Thursday (July 16) afternoon to relink.
3:22pm Fixed Rivington with a reboot
2:00pm Rivington hub is down causing connection problems in LES, East Village and Alphabet City. We are working on this now.
12:20pm Connection restored. Back to normal
12:05pm Supernode 3 has a connection problem. This is affecting all nodes in Brooklyn. We are looking into this at the moment.
3:50pm We’re back to normal now after a router reboot
11:50am We’re working on a problem affecting Brooklyn connections at the moment
10:30am The alignment problem at PH hub has been fixed and speeds should be back to normal in Brooklyn
1:30pm The storm has affected the link to our PH hub severely limiting bandwidth in the Prospect Heights and surrounding areas. We will try to fix this as soon as the rain eases.
11:45am We have fixed the connection between our PH hub and SN3. We aligned the antenna and added a radome to reduce interference. Connections to our PH hub should now be good.
Our PH hub connection to SN3 is shaky tonight affecting connections in Brooklyn, this is possibly wind related.
4:00pm Speeds from our PH sector in Brooklyn have been improved. We realigned one sector antenna today to share coverage of the northern Prospect Heights area.
Our bandwidth usage has doubled in the last week, and most connections are much slower than usual
9:16am Power problem resolved at Soft Surplus. Bandwidth should be back to normal.
11:50pm There is a power-outage at our Soft Surplus hub which is causing problems in Brooklyn. We hope to have this resolved tomorrow morning.
We are not able to get access to Saratoga until Monday. Saratoga is still online but on a backup connection.
4:00pm One of the main Saratoga connections (AF24) to Soft Surplus is down, causing slower speeds for connected nodes. We are planning on replacing a power supply in the morning.
3:30pm Fixed a power outage at 7th St which was affecting Alphabet City
2:30am Completed switching members over from s2 sector. The network is running normally now.
3:00pm Supernode 1 is back online. One of the five sector antennas (s2) failed to boot up. We will switch members over to other sectors later tonight.
7:00pm Issue with Supernode 1 at the moment, affecting Manhattan and Greenpoint. We are looking into it
3:00pm Original problem fixed. We also added a backup connection to Henry so this particular problem won’t happen again.
9:30am Our Henry St hub is having problems. This is affecting downtown Manhattan. We are working on this now.
9:00pm Our 7th St hub has been experiencing low bandwidth the past 24 hours. We are working on that now. This affects nodes in Alphabet City.
3:00pm We had a dns problem that was resolved at 5:00pm
If you are still having connections problems due to recent changes in the network please fill in the support form and mark urgent.
We’ve had a few nodes go offline because of interference. Working through some of those now. Some nodes might get stuck offline for a more extended time as we must not interfere. We’ll have to fix those after we fix all interference issues.
11:30am We are experiencing problems in Brooklyn at the moment. Sorry about this!
We are having ongoing problems with our Rivington hub and rain. We are planning to fix this within the next two weeks.
12:30pm The sector antennas at the Guernsey hub are finally working. Greenpoint connections should be ok now. Let us know via the support form if you are still having problems!
9:00pm There is still a problem with the sector antennas at the Guernsey hub
7:00pm We have finally fixed the problems with our Greenpoint Guernsey hub, caused by Wednesday night winds
Our 303 Dekalb hub is back online after wind problems from yesterday
9:00pm It is back online now and working normally
6:30pm We are having problems with the Henry and Rivington hubs. This is affecting downtown Manhattan connections
The new multipath configuration is now complete in Brooklyn and many are reporting fast speeds tonight.
There were further adjustments to the Brooklyn network today as we standardise various hub nodes. Hopefully when we add back the Soft Suplus gateway we will be back to normal bandwidth tomorrow or the next day.
2:00pm We have fixed some of the bandwidth problems. There are still some isolated slowdowns. If you are having problems please fill in a support ticket
5:00pm Bandwidth is low in Brooklyn until we fix our Soft Surplus gateway, hopefully later tonight or tomorrow.
4:30pm We have bypassed the problem and most nodes (96%) are now back online. We are still working on it.
1:40pm Our Saratoga hub is down causing outages in Brooklyn. We are looking into it at the moment.
3:30pm Upgrade completed successfully at Saratoga, and point-to-point connection established to SN4
11:30am We are performing an upgrade at Saratoga as part of an on-going effort to increase speeds. This will only affect people connected to that Brooklyn hub. The hub will be down for 30 minutes around noon and occasional ups and downs in the early afternoon.
Unusually heavy rain is interrupting connections downtown and in Greenpoint this afternoon. We have a large upgrade planned during August and September that will fix these rain problems
12:59am Bandwidth has returned to good speeds in much of Brooklyn due to improved configuration of the SN3 VPN server. This also fixes much of the Saratoga problem.
Heavy rain has been interrupting our Henry and Rivington connections downtown. This affects most nodes that are not directly connected to Supernode 1. We are planning a fix for this but it will take a few weeks to implement.
Saratoga Hub is still having bandwidth issues. We are about halfway through fixing this. There is now a fast connection from Soft Surplus to Saratoga. Next we have to upgrade the main router and also add a second West-facing sector antenna. This should all be completed over the next 4 weeks.
3:45pm Fixed the problem (actually a bad port). Connection in Brooklyn should be faster now than it has been the past week.
2:40pm Replacing a bad cable. Brooklyn nodes may be down for 20 minutes or so.
3:52pm Connection is back up. All nodes online
3:24pm Henry St hub is down affecting downtown Manhattan. We expect it to be back up within the hour
8:30pm Rivington is back online.
8:00pm The Rivington hub has disconnected causing outages downtown and at node 2090 in Greenpoint
9:38am The Saratoga hub gateway has been switched from Ehud to a 600 Mbps connection at Soft Surplus. This will improve bandwidth and latency for much of Brooklyn.
7:25pm Back to normal
6:20pm Downtown Manhattan is experiencing outages at the major hubs. We are looking into this at the moment.
11:50pm We’ve just fixed a problem at Soft Surplus that was causing slow speeds in Brooklyn since this afternoon.
2:45pm We are replacing a cable at Henry hub. This means there will be 3 minutes downtime while the router reboots. After the cable is replaced speeds from downtown hubs should increase significantly.
3:45pm: Fixed the problem, just a reboot needed. Back to normal now.
2:10pm: We have a problem at Henry hub affecting Rivington, Ave A, 7th St and Ave D hubs downtown Manhattan. Looking into this now. We should have a fix by 4:30pm.
12:10am: There was an 8 minute downtime due to an internet exchange problem. Back to normal now.
8:00pm: Things should be connected again and bandwidth should improve over the next few days.
12:00pm: We are having problems with our Soft Surplus hub node (1417) in Brooklyn since midnight. We are looking into this now. This is affecting the 9 connected nodes.
4:30pm: Normal service resumed downtown. This was a problem with integrating our new mesh configuration with some older installs.
4:25pm: Our Henry St hub is having problems and this is affecting many of our downtown nodes. We are working on fixing this now.
11:00am: We had a problem with our internet exchange for the past two hours. It appears to be back to normal now.
Five nodes connected to Ave A hub were down around 2pm today due to a configuration issue. They are back up as of 6:20pm
1pm: We rebooted the Rivington sector and it is working normally again
11am: One of the Rivington Sectors is having problems disconnecting 100Ava and about 5 connected nodes in the LES and East Village. We are working on this at the moment
Brooklyn is experiencing slow speeds especially around 8 to 9pm (“Netflix time”). We are working on fixing this.
A few people are having problems with LinkNYC kiosk using our new SXTsq VPN config. We are trying to solve this problem, but it might be better to switch back to the non-vpn config for now.
There will be a couple of short interruptions between 2pm and 3pm today as a cable is changed at Supernode 1
1:43pm maintenance completed
Due to maintenance at one of our main Brooklyn hubs there may be some temporary interruptions to between 12:30pm and 1pm today
6:00pm Power restored. All connections should be good now
5:45pm Electricians have arrived at the Saratoga building and will be working on this tonight until fixed.
9:24am There is a power outage at our Saratoga hub. This is causing problems in a large part of Brooklyn. We are sending someone out now to look at this.
DE-CIX New York, our internet exchange point is giving us new IPv4 addresses. This will cause a small amount of downtime as we switch addresses over the next two days.
Change period starts: on 2019-01-28 at 10:00 EST (15:00 UTC) End of change period: on 2019-01-30 at 07:00 EST (12:00 UTC)
Finished connecting Henry to SN1 using 60GHz gear, this upgrades Rivington connection to about 800Mbps.
Antenna alignment today at Ehud to fix ongoing bandwidth problem at Saratoga in Brooklyn. Bandwidth should improve in the afternoon after a short downtime around noon.
Rivington hub is still in process of upgrades. This is affecting 4 active nodes in the Lower East Side. We are planning to get full bandwidth there next week by connecting to Henry hub and SN1 using new gigabit 60Ghz gear.
Fixed node 1417 connection at 5:45pm
A storm took down our node 1417 and is affecting 3 nodes in the East Williamsburg area. We hope to have this fixed tomorrow.
Problem fixed! All ok.
A failed upgrade caused downtime at the major hub node in Brooklyn this morning. It was fixed at 4:15pm today.
As always, if you are experiencing issues with your connection, you can:
You can also get updates in the #1340-alerts channel on Slack.
Recently our Supernode 2 had to be taken down for roof repair which forced us to make major changes to the area’s network. We are still working on improving the new connections. The good news is many new hub nodes are coming online, so there will be more choices to connect to in the coming weeks.