NC State Shibboleth - Changes and Announcements
Maintenance Schedule
The Identity Providers will be maintained on a monthly basis to ensure we are running the latest, patched versions of the software. Upgrades will be made on the second Wednesday of each month, starting at 5pm Eastern time. These will be rolling upgrades that should not interrupt service.
Our next planned maintenance is: Wednesday, May 1, 2024 at 5:00pm - NOTE: ONE WEEK EARLY
Announcements
- The May update is being planned a week early (first Weds of the month) because of a schedule conflict on the normal date.
Change Log
April 10, 2024 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:40pm. Maintenance included the IdP 5.1.1 update with some minor interface changes, updates to Docker, and OS patching.
March 13, 2024 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to Docker and OS patching.
February 14, 2024 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:35pm. Maintenance included updates to Java, Jetty, Docker, and a minor plugin update, plus OS patching.
January 10, 2024 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates and reboots for RHEL and the jetty webserver.
December 13, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:55pm. Maintenance included updates and reboots for the RHEL 8.9 OS.
November 8, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to the docker software, java, the jetty webserver, and reboots for OS patches.
October 18, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:50pm. This was a major upgrade to Java 17, Jetty 11, and IdP 5.0.0 plus upgrades to the newest IdP plugins. The servers were also rebooted for kernel and other OS patches. Finally, the attribute release consent page was removed from the flow as permitted by our security group.
October 11, 2023 - Maintenance had to be postponed for a week.
September 13, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:35pm. Maintenance included updates to java, the jetty web server, the Duo OIDC plugin, and the docker software, and reboots for a system patches. Some old custom login code for the Brickyard servers was also removed as it is no longer being used.
August 9, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to java and the docker software, and reboots for a kernel patch. The Jetty webserver and load balancer were updated to allow longer cookie headers as well, to mitigate the number of random 400 errors seen by some users.
July 31, 2023 - The SSL certificates for shib.ncsu.edu were renewed today and pushed out to the load balancers. The IdP servers were also restarted one at a time to run with the new cert internally.
July 12, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to the docker software, and reboots for a kernel patch.
June 14, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to the OIDC/duo plugins and the docker software, and reboots for a kernel patch. The CSS was updated to start using the new brand font Roboto in place of Univers.
May 10, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to the docker images to base them on rocky 8, plus software updates to java, the jetty web server, and docker.
April 12, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to IdP version 4.1.3 and to docker, and reboots for a kernel patch. There was also a minor code change to better handle IdP-side authentication flows.
March 8, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to java, the jetty web server, and docker, and reboots for a kernel patch.
February 8, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included a minor upgrade to IdP 4.3.0, a major version upgrade to Jetty 10.0.13 and to Docker CE 23, and a patch to the login page to try to prevent accidental double-clicks on the log in button. The logout redir page was also patched so it could not be used as an open redirect. All servers were rebooted to get them on the same, most recent kernel.
January 11, 2023 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:10pm. Maintenance included updates to docker. Servers were recently rebooted for NFS maintenance on Jan 2.
December 14, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:10pm. Maintenance included updates to the jetty web server and docker, and an upgrade to the Duo plugin to enable additional logging.
November 9, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:35pm. Maintenance included updates to java and docker, and reboots to load the latest kernel patches.
October 24, 2022 - Unity login servers for shib.ncsu.edu were restarted to revert the temporary update from October 12th.
October 12, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00 and 5:15pm. This update included minor patches, and a temporary update for the data center power outage scheduled for the weekend of Oct 15 - Oct 17.
September 28, 2022 - The shib.ncsu.edu servers were not changed today. The data center power outage planned for this weekend will be rescheduled.
September 14, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to the Java and Docker, and reboots to load the latest kernel patches. The local LDAP server was flushed and rebuilt for an attribute update coming soon.
August 10, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:50pm. Maintenance included updates to the Java and Docker, and reboots to load the latest kernel patches. The shib.ncsu.edu SSL certificate was renewed and replaced on the servers and the load balancers.
August 3-4, 2022 - Host servers that run the docker containers were replaced and upgraded to RHEL 8. All restarts were rolled in to the load balancers to ensure no sessions were interrupted.
July 13, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to the IdP software libraries, the Jetty web server, and reboots to load the latest kernel patches.
June 30, 2022 - The shib.ncsu.edu servers were restarted this afternoon to revert the temporary update from last week. The restarts ran between 5:00 and 5:10pm.
June 24, 2022 - The shib.ncsu.edu servers were restarted this afternoon to load a temporary update for the IdM maintenance scheduled this weekend. The restarts ran between 5:00 and 5:10pm.
June 8, 2022 - The monthly maintenance was completed on time after a bug was found and patched. Reboots ran between 5:00pm and 5:30pm, and the bug was fixed by 6:00pm. This update included a rewrite of the custom SP configurations, a new version of Docker, and reboots to load the latest kernel patches.
May 5, 2022 - The monthly maintenance for May was completed on morning of Thursday May 5th. Services were impacted between 7:00am and 7:45am. This update deployed the new Duo Universal Prompt flow and an upgrade to IdP version 4.2.1.
April 13, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to the IdP software, the Jetty web server, and Docker, and reboots to load the latest kernel patches.
March 9, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Java and Docker, removal of the code for user alert page, and reboots to load the latest kernel patches.
February 9, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to Java, IdP 4.1.5, the Jetty web server, and reboots to load the latest CentOS kernel patches.
January 12, 2022 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included a minor updates to the OS packages and Docker, and reboots to load the latest RHEL kernel patches. The emergency change implemented on Jan 10 was reverted since the underlying problem has been resolved.
January 10, 2022 - An emergency configuration change was pushed out to the shib.ncsu.edu servers between 11:35 and 11:45am. This implemented change ticket: CHG0031668. Users may have seen a brief interruption during login for the restarts.
December 8, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included a minor updates to the OS packages and Docker, and reboots to load the latest kernel patches.
November 30, 2021 - The Shibboleth team released SP v3.3.0 this morning with some minor patches. The updated packages were published to the CLS and vision4 repos for RHEL/CentOS Linux 7 and 8.
November 10, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included a minor updates to the OS packages, Docker, and Java, and reboots to load the latest kernel patches.
October 13, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included a minor updates to the OS packages, Docker, and Jetty, and reboots to load the latest kernel patches.
October 8, 2021 - Evening - InCommon announced that their problem has been fixed. Our IdP servers were restarted again between 5:45-5:55pm to remove the temporary /etc/hosts entries. The service appears to be running properly.
October 8, 2021 - Morning - InCommon continues to have problems with their MDQ service, posted here: InCommon Outage 2021-10-07. We have followed their suggestion to add /etc/hosts entries to each of the IdPs to point the MDQ hostname to a known-working CloudFront IP address. Those changes were made with server restarts between 9:30 and 10:05am. The missing SP entities are resolving again, at least for now.
October 7, 2021 - This morning a number of SP entities are missing / not resolving in the InCommon federation MDQ feed. Two of those SPs, used for jobs.ncsu.edu and apply.ncsu.edu, were manually re-added as local metadata to ensure those services could continue to login. This required a restart of shib.ncsu.edu that was done between 10:30 and 10:40am. We have an open ticket with InCommon to resolve the other SP entities that are still missing.
September 8, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included a minor updates to the OS packages and reboots to load the latest kernel patches. The SSL certificate for shib.ncsu.edu was also renewed.
September 1, 2021 - A rolling update to the shib.ncsu.edu servers was done to load a new user alert page. Servers were reloaded between 5:00 and 5:30pm.
August 30, 2021 - A configuration change that has been developed for an update on Wednesday was accidentally published to the production on-campus shib.ncsu.edu servers. This resulted in user being unable to login for a period of about 20 minutes before the change could be reverted. Users on the off-campus servers at AWS were not affected. The outage ran from about 14:10 to 14:30.
August 13, 2021 - Did a rolling update on the shib.ncsu.edu servers to add a new attribute to the ldap schema. Services were impacted between 5:00pm and 5:25pm. Only the local ldap servers were restarted, but the IdP's had to be removed from service while the data reloaded.
August 11, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included a minor updates to the IdP software, Java, and Docker, and reboots to load the latest kernel patches.
July 14, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included a minor update to the Jetty web server, and reboots to load the latest kernel patches.
July 13, 2021 - The Shibboleth team released SP v3.2.3 last week with some minor patches primarily for Windows servers. The updated packages were published to the vision4 repo for RHEL/CentOS Linux 7 and 8.
June 9, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included a minor updates to IdP 4.1.2, to the Jetty web server, to the Docker container system.
May 30, 2021 - On Sunday morning at 8am we ran a special maintenance to make some changes to visual elements on the login screens. The new belltower logo replaced the current block-S logo on Unity login pages. There were also some changes to the sidebar text, and the addition of a "show password" checkbox on login screens.
May 12, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included minor updates to the Java runtime, to the Jetty web server, and a reboot for kernel patches.
April 14, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included an update to the IdP to 4.1.0, minor updates to the Jetty web server and Docker, and a reboot for kernel patches.
March 10, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:10pm. Maintenance included minor updates to the Jetty web server and Docker.
February 23, 2021 - The Zoom SP updated their metadata to a new cert that was not being used on our account. IdP servers that loaded this new metadata were unable to allow users to login. 3/4 of the AWS servers and 1/6 of the on-campus servers were affected. DELTA fixed the zoom account to use the correct certs, and we restarted the 6 IdP server that were still trying to use the old metadata. Zoom logins were affected around 1:15pm to 3:00pm, and the restarts were done just before 3:00pm.
February 10, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to Java 11, Jetty, Docker, and reboots to load the latest kernel on all servers. There was also a minor URL change for the logo used on the Unity login page, but the image did not change.
January 13, 2021 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included the delayed update to Docker CE 20, and reboots to load the latest kernel on all servers.
December 16, 2020 - The Shibboleth team released SP v3.2.0 yesterday with some minor new features and a security patch. The updated packages were published to the vision4 repo for RHEL/CentOS Linux 7 and 8. Version 6 is no longer supported.
December 9, 2020 - Monthly maintenance completed with one omission. Services were imacted between 5:00pm and 5:15pm. Maintenance included an update to the Jetty webserver and a reboot to load a new kernel. A planned minor update to docker-ce 19.x was skipped because Docker released version 20 in our upstream, and that version did not appear to be working correctly on a quick test. That update will be tested and hopefully deployed in January.
November 11, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to Java and to the Jetty webserver, which fixed client IP bug observed last month. The RHEL servers were also rebooted to load a new kernel.
October 14, 2020 - Monthly maintenance was not completed as planned. There were Docker updates and server reboots for kernels and those were completed. The new image contained an update to the Jetty webserver, and a change to the main jetty config file that appeared to be working correctly in testing. When used behind the load balancers, it became obvious that we were now logging the LB IP addresses and not the actual client IPs. All production servers were reverted to the previous image build until this update can be tested further. Services were affected from 5:00pm to 5:20pm with no service outage.
October 2, 2020 - The SSL certificate for shib.ncsu.edu was renewed prior to expiration which would have occurred before the next maintenance. The public facing load balancers were updated around 6:40am, and the back-end servers were updated and restarted one at a time from 7:00 to 7:15am. There was no service outage.
September 22, 2020 - Re-added EPPN as a default attribute on the Brickyard IdP servers on affil.shib.ncsu.edu.
September 9, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:10pm. Maintenance included a reboot to load a new kernel.
August 12, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:10pm. Maintenance included updates to Java and the Jetty webserver.
August 10, 2020 - The production IdP servers were rebooted around 5pm to fix a problem with the AFS client and to load a new kernel.
July 29, 2020 - The IdP servers were restarted between 5:00 and 5:05pm to load some changes to the sidebar content for Duo and to update the footer links on all pages.
July 8, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included the upgrade to IdP version 4.0.1 and Java 11, plus minor patches to to the Jetty webserver, Docker, and a reboot for a kernel patch.
June 20, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to the Jetty webserver, Docker, and a reboot for a kernel patch.
May 29, 2020 - The IdP servers were restarted between 5:00 and 5:10pm in order to update the SSL chain certificates that were presented to validate the InCommon CA. The root CA in that chain expires May 30th. The servers are now presenting a new chain with a longer-life CA.
May 10, 2020 - Monthly maintenance completed as planned. Services were impacted between 8:00am and 8:10am. Maintenance included updates to Java, the Jetty webserver, and a reboot for a kernel patch. We also updated our metadata and login pages as a part of the NC State Login Changes.
May 8, 2020 - One of the AWS login servers crashed on Friday night and had to be rebuilt to bring it back online. The remaining servers in the pool handled the load with no issues. The server was back online by 3:30pm on Sat, May 9.
April 8, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Docker and a reboot for a kernel patch.
March 11, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to Docker and Jetty. We also converted the InCommon federation load to use MDQ to save memory on the server.
February 12, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Java and Jetty, and a reboot for a kernel patch. There was also a configuration update to enable htmlLocalStorage for IdP session data.
January 8, 2020 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included only minor updates and a restart of the services.
December 11, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to Jetty and Docker, and a reboot for a kernel patch.
November 13, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Jetty, Java, and Docker, and a reboot for a kernel patch.
November 11, 2019 - Parent and Guest logins were having occasional errors while logging in to affil.shib.ncsu.edu since Oct 29 or Nov 7. Our AD provider was closing connections in a shorter time than previously expected. We adjusted the validation time and the problem has been resolved.
October 9, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included updates to Jetty and the IdP software, and a reboot for a kernel patch.
September 15, 2019 - At 7am on Sunday, we changed the DNS entries for shib.ncsu.edu and affil.shib.ncsu.edu to point to the new servers running in AWS and using Azure for AD. This change will only affect off-campus DNS queries. On-campus users will continue to use on-campus login servers.
September 11, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:40pm. Maintenance included updates to Jetty and Docker, and a reboot for a kernel patch.
August 14, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Java OpenJDK and Docker, and a reboot for the RHEL 7.7 kernel patch.
July 10, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Java OpenJDK, the Jetty webserver, and Docker, and a reboot for a kernel patch.
June 5, 2019 - Monthly maintenance was run one week early this month, and completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to docker and a reboot for a kernel patch.
May 8, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to Java, the Jetty webserver, and the IdP software, and a reboot for a kernel patch.
April 10, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:35pm. Maintenance included an update to the Docker software, the Linux OS, and a reboot for a kernel patch.
March 14, 2019 - Federation metadata updates were published today to remove the old IdP certificate from our metadata. NCSU Fed was updated around 6:00am. InCommon Fed will update around 3:15pm. UNC Federation update has been requested as of 7:30am.
March 13, 2019 - As part of our regularly scheduled monthly maintenance, we have changed the signing key on our shib.ncsu.edu IdP. For more details, see the Key Update March 2019 page. Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. In addition to the key change, we updated the Jetty webserver software.
March 11, 2019 - The Shibboleth team released SP v3.0.4 today with some bug fixes and a security patch. The updated packages were published to the vision4 repo.
February 20, 2019 - We are preparing to change the signing key used by our shib.ncsu.edu IdP in March. We have changed our IdP metadata distributed by the NCSU, UNC-GA, and InCommon Federations to include the new signing certificate in addition to the old one. For more details, see the Key Update March 2019 page.
February 13, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included an update to the IdP software to version 3.4.3, an update to Java, and renewal of the signing certificate used by the affil.shib Parent/Guest system IdP. Tests of SPs on the guest system suggest that it is still running correctly.
February 11, 2019 - The public certificate for the NCSU Federation signing key has been re-signed for another 10 years. The key has not changed. The first metadata using that cert in its signature was published this morning. Additionally, the metadata with the updated certs for IdPs: affil, idpt1, idpt2, affilt1, and affilt2 has been released. The affil servers will continue using the existing key and the new cert on Weds, Feb 13 after the monthly maintenance. The test servers have all been updated today.
January 10, 2019 - Our Shibboleth login services were degraded from around 4:00pm to 5:40pm due to a service degradation from Duo Security. We implemented a workaround starting at 4:45pm and restored normal service after the outage around 5:40pm.
January 9, 2019 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included an update to the IdP software to version 3.4.2.
December 19, 2018 - The Shibboleth team released SP v3.0.3 today with some bug fixes. The updated packages were published to the vision4 repo.
December 12, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. Maintenance included an update to the Jetty webserver and reboots for kernel patches.
November 14, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included an upgrade to the IdP 3.4.1 software and configuration files, an update to Java, an update to RHEL 7.6 on the servers, and a reboot to run the new kernel.
October 25, 2018 - We found a bug in the metadata submitted by one of our new SPs in the NCSU Federation. The XML code was causing a validation failure on sites running SP 3.x which meant that new metadata was not being loaded. The bad SP was corrected and our import process will now check for this error on future imports.
October 10, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included a reboot to run the latest host OS kernel.
September 20-26, 2018 - During this week we upgraded the host servers to our latest linux / docker kit. There was no loss of service as only single servers were reloaded at one time.
September 19, 2018 - Monthly maintenance completed one week late due to hurricane Florence. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to the Jetty server, the Docker service, and a reboot to run the latest host OS kernel.
September 19, 2018 - We moved the shib.ncsu.edu and affil.shib.ncsu.edu addresses to the Fortinet load balancer between 6:00 and 6:10am.
September 18, 2018 - Our Shibboleth login services were degraded from around 9:15am to 10:00am due to a service degradation from Duo Security. Users who require 2FA to login reported that Duo was not loading the iframe to complete their requests. We implemented a workaround starting at 10am, and restored normal service after the outage around 12:20pm.
August 29, 2018 - Our Shibboleth login services were degraded from around 10:12am to 10:40am due to a service outage from Duo Security. The preauth checks were not responding from Duo causing Shibboleth to try to send everyone to use a Duo login, whether they were enrolled or not. We implemented a workaround starting at 10:40am, and restored normal service after the outage around 1:40pm. We have also added addtional code to detect this kind of preauth failure and handle it better in the future.
August 18, 2018 - Our Shibboleth login services were degraded from around 11:20am to 12:00pm due to a service outage from Duo Security. Users who require 2FA to login reported that Duo did not respond to their 2FA requests. We implemented a workaround starting at 12 noon, and restored normal service after the outage around 1:35pm.
August 14, 2018 - Updated the vision4 repo with Service Provider v3.0.2 packages. Notes and instructions are available on the Service Provider V3.0 Upgrade page. We also updated the installation notes to describe SP V3.0 installation.
August 13, 2018 - 8:45am - The test login server affilt2 was restored to service.
August 8, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to the Java libraries, the Docker service, and a reboot to run the latest host OS kernel.
August 6, 2018 - 11:30am - The test login server affilt2 is offline due to an expired SSL certificate on the AD server that it uses. We have a ticket in with the AD team to get it fixed. We will restart the server as soon as we can.
July 11, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. This update included configuration updates to fix a security problem and to allow us to publish isMemberOf as a new attribute.
June 13, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to the Jetty webserver and a security patch to the IdP software to version 3.3.3. We also enabled support for requests that require a Refeds MFA login context. The Affiliates IdPs were patched to make minor changes to the text on the login screen. Both IdPs were patched with a slightly different error message on a failed login attempt.
May 29, 2018 - Minor configuration update to evaluate InCommon Federation metadata before the UNC-System Federation metadatas. This is intended to solve a problem with an InCommon partner that has the same SP entityID registered in both and the wrong version is being used by our servers. The IdPs were restarted at 5pm to implement this update.
May 9, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to the Java libraries. The host servers were updated to RHEL 7.5 and the latest Docker. We also implemented a security patch on the attribute release consent page.
April 11, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to the server OS and the Jetty webserver. The RHEL 7.5 update released yesterday was not included.
March 28, 2018 - Our IdP servers were unable to complete logins from about 15:15 to 16:00 due to a mistake with the internal IP addresses used to reach our AD servers.
March 14, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to the server OS, some minor configuration changes to address SSL security, and a reboot for kernel patches.
February 27, 2018 - Built and released the XMLtooling 1.6.4 package in the vision4 repo. This is a security patch as announced in secadv_20180227 by the Shibboleth team.
February 14, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to the server OS and Java, and a reboot for kernel patches.
January 22, 2018 - We have been getting an increasing number of reports from users who are able to login to our IdP but who do not pass attributes along to the Service Provider. This is due to our AD connectors timing out when trying to retrieve their attributes. We have increased the timeout on these connections from 3s to 10s to see if that will mitigate the issue.
January 16, 2018 - Built and released the XMLtooling 1.6.3 package in the vision4 repo. This is a security patch as announced in secadv_20180112 by the Shibboleth team. They stated this is a critical patch for Service Providers on RHEL / CentOS 7 servers as there is an exploit in the wild.
January 10, 2018 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance only included a reboot for OS patches this month.
December 13, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. Maintenance included updates to the server OS and Jetty, a reboot for kernel patches, and an update to the Duo/2FA blurb on the login page for Unity logins.
November 30, 2017 - The Shibboleth team announced a security patch to the curl-openssl package that is included with the Shibboleth SP software for linux and windows. The updated linux RPMs have been added to the vision4 repo.
November 17, 2017 - The Shibboleth team announced a followup patch_20171117 to the xmltooling library used by the SP 2.6.1 software release earlier. Service providers are advised to update the library and then explicitly restart their shibd service to ensure the patched code is running correctly. The updated xmltooling library has been added to the vision4 repo.
November 15, 2017 - Built and released SP 2.6.1 packages in the vision4 repo. This is a security patch as announced in secadv_20171115 by the Shibboleth team. The feature that was patched is not one that we commonly use at NCSU.
November 8, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. Maintenance included updates to the server OS, Java, and a reboot for kernel patches.
October 11, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included updates to the server OS, Jetty, IdP 3.3.2 security patch, and a reboot for kernel patches.
September 13, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. Maintenance included an updated SSL certificate for shib.ncsu.edu, along with server reboots to load new kernel software on the login servers.
August 23-24, 2017 - One of the four servers hosting docs.shib.ncsu.edu received a bad update that prevented it from running PHP scripts correctly. Users may have had trouble reaching the WAYF and /duoenroll/ sites during the outage: 5:00pm on Aug 23 through 1:30pm Aug 24.
August 23, 2017 - We restarted all of the IdP servers to load the InCommon SSL certificates to be used by Active Directory. The previous AD key change was delayed to a future date. This update also changed the password expiration warning page such that it now includes the date that the password will expire.
August 18, 2017 - Login services were interrupted from about 2:10- 2:25pm today due to a bad configuration update that was pushed to the login servers. The configs were reverted and the servers resumed normal operation.
August 9, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. This upgrade included a Java update, an update to the docker server software, and a reboot for kernel patches.
August 2, 2017 - We restarted all of the IdP servers to load the new NCSU CA certificate used by Active Directory. Those servers will be switching keys on Aug 9 at 1pm, a few hours before the next maintenance date. -- The AD key change was delayed due to errors in testing the new keys.
July 12, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:30pm. This upgrade included minor updates to the OS and kernel. Two new user attributes were added to the default NCSU Federation release policy. These are documented on the Two-Factor Attributes page.
July 7, 2017 - We were alerted to a security bug in our IdP code on shib.ncsu.edu. The bug was patched and the servers were updated and restarted from 7:00-7:30am.
June 27, 2017 - The login service experienced an outage starting around 10:00pm on Monday night until around 01:15am this morning. There were power problems in our datacenter that caused our load balancers to lose sync with each other. Neither of them were routing the shib.ncsu.edu host until the servers were restarted and resynced.
June 21, 2017 - The internal ldap service that we maintain for our user attributes has been updated to add two new attributes to the NCSU schema.
June 14, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. This upgrade included updates to the OS, Java, and Jetty. We upgraded to IdP 3.3.1 which includes a significant change to the Duo login flow, but it should be transparent to our users. We also added a notice to the main login page to encourage Duo enrollment.
May 18, 2017 - The internal ldap service that we maintain for our user attributes has been updated to look for and then load changes five times each day instead of one. This will allow us to be more responsive to changes made by our new IdM system.
May 10, 2017 - No monthly maintenance due to the OIM Go-Live.
May 9, 2017 - Shibboleth attributes start using feeds from OIM system.
April 12, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:35pm. This upgrade included minor updates to the OS and Jetty. A small change was made to the information found on the Duo login page to provide links to additional help with Duo. In addition, the shib.ncsu.edu VMs were upgraded increase their memory allocation.
April 7, 2017 - After a week of random IdP server crashes, we found that the current InCommon metadata set is too large for our Java heap memory limits. We restarted the shib.ncsu.edu Idp servers between 8:15 and 8:45 this morning to increase that limit.
March 8, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. This upgrade included a Jetty update from 9.3.x to 9.4.x, an update to the docker server software, and a reboot for kernel patches.
February 8, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. This upgrade included minor updates to Java and Jetty and a reboot for kernel patches and a docker update. In addition, we removed the "Attention: Welcome to the new look..." block from the login page.
January 11, 2017 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm. This upgrade included a minor Jetty update only.
December 14, 2016 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:20pm. This upgrade included a minor Jetty update and a reboot for new kernel patches.
November 15, 2016 - UComm requested a few more changes to fix accessibility issues with our CSS styling. The servers were restarted this morning to deploy those fixes.
November 10, 2016 - A CSS bug was found such that Safari browser users were not seeing the dots in the password field when they typed in their credentials. This was fixed and the patched image was pushed out to the servers.
November 9, 2016 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm. This upgrade included changing the page themes to match UComm's suggestions for branding, and the addition of stricter SSL settings. For full details see the SysNews post about removing TLS 1.0/1.1 support.
November 7, 2016 - Our remaining IdP v2 clients have been updated to use IdP v3. The idpv2.shib.ncsu.edu service has been terminated.
October 27, 2016 - The Shibboleth Advisory on 27 October 2016 could potentially affect our service. The LDAP cache has been disabled as recommended on shib.ncsu.edu. The service will load the update without a restart or outage.
October 12, 2016 - Monthly maintenance was canceled due to an unexpected schedule conflict. The Jetty update and TLS settings changes will be made in November.
September 16, 2016 - We had an error in one of our upstream data sources that tried to drop the student affiliation from 30k of our 38k student accounts. Our systems correctly refused to load the update at 8:20am. Logins were not affected as they just continued to use affiliations from the previous day's load. The data was corrected by 1:30pm and is being loaded into the internal LDAP servers one by one.
September 14, 2016 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:15pm.
September 7, 2016 - We added Duo two-factor support to our Unity IdP shib.ncsu.edu. Anyone with an enrolled Duo account will be asked to complete their login using their two-factor application. Users without Duo accounts will continue to login with just their password.
We also included a fix to the IdP session timeout times. The IdP login sessions will timeout after 1 hour of inactivity or 10 hours total lifetime, as was originally intended. Current default timeouts are 30 minutes inactivity or 1 hour total lifetime.
August 10, 2016 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm.
July 26, 2016 - Even after the LDAP servers reported they had reverted, we were still getting reports of some users getting inconsistent affiliation results. We did a rolling restart of all Unity IdP servers starting at 1:15pm and ending at 1:35pm, to clear all the caches.
July 26, 2016 - Our upstream data dropped an important table which caused our LDAP data to lose track of many affiliations. We reverted our LDAP to a backup copy from yesterday, effective at 9:23am today.
July 13, 2016 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:35pm.
June 30, 2016 - Built and released SP 2.6.0 packages in the vision4 repo and the vision3 repo. Vision3 has reached EOL and will not receive any additional updates.
June 9, 2016 - The Jetty patches in the last update disabled all protocols except TLS1.2. This was not an intended update at this time. We have rolled back the servers as of 11:45am.
June 8, 2016 - Monthly maintenance completed as planned. Services were impacted between 5:00pm and 5:25pm.
June 3, 2016 - The v2 server for IdP test 1 has been removed. The v3 servers idpt1.shib.ncsu.edu and affilt1.shib.ncsu.edu have been added to replace it. These are currently configured for use with OIM testing.
May 29, 2016 - IdP test 2 server has been reconfigured to support Duo two-factor authentication, for those with Duo accounts.
May 11, 2016 - Successful Go-Live for IdP v3 at 7:00am as announced below. A few Service Providers were allowed to continue to use IdP v2 while they work out compatibility problems with old libraries.
May 9, 2016 - SPs using Affiliates login switching to affil.shib.ncsu.edu starting at 9am.
March 16, 2016 - Announcement for Upgrade to IdPv3 on May 11 posted to SysNews.
February 26, 2016 - Built and released SP 2.5.6 packages in the vision4 repo and the vision3 repo.
February 23, 2016 - Split IdP test2 server into two new entities. These servers will be used for IdPv3 testing.
- idpt2.unity.ncsu.edu/idp is now idpt2.shib.ncsu.edu/idp
idpt2.unity.ncsu.edu/other-idp is now affilt2.shib.ncsu.edu/idp
Feb 24 Note: v2 idpt2.unity.ncsu.edu/other-idp had to be restored, it was still being used for development.
February 18, 2016 - IdP server reboots for OS security patches
January 8, 2016 - Fixed logo and info links in all metadata files.
January 6, 2016 - Placed redirects on shib.ncsu.edu servers for all non-IdP traffic.
December 2, 2015 - Reconfigured IdP servers to use docs.shib URLs for federation metadata.
November 30, 2015 - Deployed separate docs.shib host dedicated to hosting these documentation and federation files.
October 15, 2015 - Reconfigured IdP servers for better LDAP caching. Should reduce the recent failures due to LDAP overload.
August 11, 2015 - Built and released updates to xmltooling-1.5.6-1.1 and shibboleth-2.5.5-3.1 SP packages.
July 21, 2015 - Built and released SP 2.5.5 packages in vision3 repo.
March 20, 2015 - Built and released SP 2.5.4 packages in vision3 repo.