Product:

AppAssure 5.x

Problem Description:

After resolving an Unknown WMI Error on SQL attachability checks following the process in this KB http://www.appassure.com/support/KB/unknown-wmi-error-invalid-class/
The same error recurs nightly. It can be temporarily resolved, but still recurs.

Problem Cause:

This is caused by the WMI being rebuilt by Configuration Manager in Windows Management Framework.

Resolution:

Setting the registry key HKLM\SOFTWARE\Microsoft\CCM\CcmEval\NotifyOnly to TRUE
will prevent Configuration Manager from rebuilding the WMI repository even if it incorrectly detects corruption.
See this aricle for documentation. http://social.technet.microsoft.com/Forums/en-US/00f7fa7c-a7a6-490e-af20-d8cc13a788f0/having-to-run-mofcomp-fix-after-every-restart?forum=dpmsqlbackup
This issue is fixed in System Center 2012 Configuration Manager SP1.

Product:

Replay 4.x

Problem Description:

When attempting to explore a recovery point, the following error comes up: “is not a valid IP address Parameter name: targetIP”.

Problem Cause:

The admin console is not reading the proper IP address from the core server.

Resolution:

1. Select the Summary tab of the core server in the admin console. There should be an IP address listed. The Core server name should also have an IP address in parenthesis; if this is missing you may see this error.
2. After verifying the IP address displayed in the admin console, check the IP address for the core server – in the command prompt, run ipconfig /all.
3. Also, check that the NIc order is correct. Replay will use the first connection listed. To verify this, go to Network Connections, press ALT and then in the menu bar click Advanced, then select Advanced Settings. Make sure the first network connection has the IP address you with the core to use assigned to it.
4. After verifying that this is all correct, close the admin console, then make sure the replayadminconsole.exe process ends. Kill that process if necessary.
5. Open the admin console again, and the correct IP address should be displayed, and recovery points should be mountable.

Product:

AppAssure 5.x

Problem Description:

“System.InvalidOperationException: Could not retrieve type from registry” is present when attempting to open either the Replication tab or Machines tab. “Unable to start the replication communication service” error is displayed.

Problem Cause:

This is caused by extra or corrupted registry keys in the HKLM/Software/AppRecovery/Core/Replication branch.

Resolution:

1. Back up the registry keys
2. On the target core, delete the source core from replication, but do not select the option “with recovery points.”
3. Browse to HKLM/Software/AppRecovery/Core/Replication on the source core server.
4. Delete this key and all its subkeys.
5. Repair the AppAssure core from Programs and Features.
6. Once the repair is complete, you will be able to access the Machines tab and Replication tab in the core console.
7. Re-add the remote core to replication.

Product:

AppAssure 5.x

Description:

Copy speeds from a mounted recovery point in AppAssure can be noticeably slower than copying from one disk location to another.  This article details several reasons and causes to slow transfer speeds.

Problem Cause:

Copy performance is affected by the following factors:
- Copying from a mounted recovery point is slower than disk-to-disk because the point is presented by the virtual disk service
- Fragmentation of the original data, number of individual files in a restore job, and the amount of deduplication in the repository all affect the speed of copying as well.
- Copying multiple files at once will slow down overall performance.
- Hardware, most especially the repository storage device, is a major factor in restore speeds. High disk IO capability, I/O Operations per second, SpinDown Rates, and network latency, are key to ensuring less overhead on performance.
- If copying from the core server to another server, local networking speeds and congestion will also be a factor.

Resolution:

We can use the Local Mount Utility to mount recovery points on the protected server, and while we will still be affected by repository performance and the limitations of the virtual disk service, we can eliminate having to copy across the network.
In critical cases, pausing other core activity will reduce the load on the repository.

Products:

AppAssure 5.3, Replay 4.x

Problem Description:

After mounting a recovery point, you are unable to browse the mounted volume or volumes. Taking ownership of the volume from the Security tab succeeds but you are still unable to browse the volume due to permissions errors.

Problem Cause:

UAC on the core server is not allowing complete ownership of the volume.

Resolution:

1. Turn off User Account Control on the core server. To do so:
2. Go to control panel>User Accounts>Change User Account Control Settings
3. Lower the slider all the way
4. Reboot the server to complete the process.
5. Then remount the recovery point in question, using the Writable option. You should be able to browse the recovery point and take ownership of any directories or files which have special permissions applied to them.

Product:

AppAssure 5.x

Problem Description:

Client is unable to map a repository connected to his Windows storage server. Receiving errors:
““Unable to get amount of a free space available on \\Servername\repository – The handle is invalid””
““User does not have write permissions for the directory \\ServerName\e\respository
At
Replay.Core.Implementation.Repositories.RepositoryService.ValidateWritePermissionsForNetworkShare(IEnumerable`1 fileSpecifications)””

Problem Cause:

The repository is not mounting due to the Windows Storage Server 2008R2 comes with preconfigured folder “serverfolders”, under which has nested folders. You can create folders in each partition but AppAssure will not add a repository on the partitions unless it’s pointing to \\ServerName\ServerFolders.

Resolution:

1. Open the Core console
2. Select configuration tab then select Add repository
3. Type UNC path \\ServerName\ServerFolders\desired_folder along with user name and password
4. Select add

Product:

AppAssure 5.x

Description:

Core UI is unresponsive on a Replication Target Core after adding 2-3+ Source Cores to Replication.

Resolution. 

Make these changes on the replication target core and all source cores. Performance should be seen to improve once all cores have been changed and replication is re-enabled.  For 6+ Replication Source, change the decimal rate to 3-4 times the original.

HKEY_LOCAL_MACHINE\SOFTWARE\AppRecovery\Core\serviceHost\RequestProcessorConfiguration
- double the decimal value for ConcurrentCheapRequests
- double the decimal value for ConcurrentExpensiveRequests

HKEY_LOCAL_MACHINE\SOFTWARE\AppRecovery\Core\webServer\RequestProcessorConfiguration
- double the decimal value for ConcurrentCheapRequests
- double the decimal value for ConcurrentExpensiveRequests

Stop and then start the core service.

Description

When installing AppAssure software it will get stuck on “Preparing Installer”
When checking Task Manager -> Performance; there is no activity

Solution

Go into the Control Panel
Click on User Accounts
Click on Change the User Account Control Settings
Move the slider bar to the bottom – Never Notify
Reboot the server
Log back in and run the Installer again.
(Note: You can bring up Task Manager -> Performance, to watch activity as it unzips the files)

Product:

AppAssure Core 5.x

Problem Description:

Customer is unable to perform Rollups because they are receiving error: Sequence contains more than one element. This error occurs during Nightly jobs and forced Rollups.

Problem Cause:

Issue occurred during nightly jobs-Rollups

Resolution:

1. Run Chkdsk /r /f on the storage repository.  This will require a reboot of the system.
2. Run a Full Repository Check

To run a repository check:
1. Select the Core Name.
2. Select Configuration.
3. Select Repositories.
4. Check the box next to Expand the Repository.
5. Click Check
6. Let nightly jobs run.

Products:

AppAssure 5.x

Replay 4.x

Problem Description:

Trying to start Agent service results in an unknown WMI error – “WMI unauthorized access exception”

Replay.log error:
ERROR 2013-10-09T16:59:03 [4] – Replay.Agent.Service.AgentService ()
Caught exception during service start
Replay.Common.Contracts.WmiApi.WmiException: Unknown WMI error: —> System.UnauthorizedAccessException

WMIDiag results in:
0×80070005 – (WBEM_UNKNOWN) This error code is external to WMI.
=> This error is not a WMI error. It is typically due to: The DCOM security
modifications.
=> Ensure that DCOM security configuration settings are not modified.

Problem Cause:

This issue is caused by restrictive DCOM permission settings.

Resolution:

1.) Use Component Services (dcomcnfg.exe) to change the security if the “Edit
Limits” button is not grayed out:

Console Root/Component Services/Computers -
right click on “My Computer” and select Properties. Go to COM Security,
Launch and Activation Permissions. (Since the “Edit Limits” button was
grayed out, no changes were made here)

2.) If “Edit Limits” is greyed out, please check Local Security Settings (secpol.msc) for any DCOM policies:
Security Settings/Local Policies/Security Options: DCOM: Machine Launch Restrictions in Security Descriptor Definition Language
(SDDL) syntax

3.)Double click (or right click, properties), Edit Security, Add “Everyone” &
check:
Local Launch and Local Activation..

Product:

AppAssure 5.x

Problem Description:

When installing AppAssure software it will get stuck on “Preparing Installer”
When checking Task Manager -> Performance, there is no activity

Resolution:

Go into the Control Panel
Click on User Accounts
Click on Change the User Account Control Settings
Move the slider bar to the bottom – Never Notify
Reboot the server
Log back in and run the Installer again.
(Note: You can bring up Task Manager -> Performance, to watch activity as it unzips)

Product:

AppAssure 5.x

Problem Description:

Agent showing offline or Core is reporting agent is already protected

Problem Cause:

Error is received when an agent is offline. If you unprotect and re-protect the agent you receive the error “Name already exist on the core”

Resolution:

1. Stop all jobs on agent (if re-added under another name after failure)
2. On the agent system and stop the service
3. In the core console, remove agent from protection (if re-added under another name after failure)
4. On the Agent, go to regedit
5. Delete agent pairing settings under (HKEY_LOCAL_MACHINE\Software\apprecovery\agent)
6. On the Core, select agent under recovery points only list
7. Copy agent ID in the web URL
Example: https://name:8006/apprecovery/admin/ProtectedMachine/e1a2866d-ac75-4c50-8c69-16cf10935e9c/Details
8. On the Agent system , open regedit and drill down to: (HKEY_LOCAL_MACHINE\Software\apprecovery\agent)
9. Open agent ID and select the registry entry for Agent ID
10. Right click and modify Agent ID
11. Paste the Agent ID from the web URL into agent ID field and select ok
12. Start agent service
13. On the Core console, select protect machine
14. Enter agent info and protect

Products:

Replay 4xxx (All currently available releases)

Description:

When the agent is installed on the core and the self protected cores snapshots are taken you may sometimes receive the Incorrect Function Error and the associated backup process will fail

Thread: 5772

Logger: tevoSource.snapshotHandler
Context: PG=VolsWithInterval60Minutes Volume=\?Volume{f67450f5-e34c-11de-8c72-0026b952626c} DriveLetter=E:
Source Location: SnapshotHandler.cpp:992
Details:
Transfer failed on volume: E: Epoch: 1508 Target: TITAN-BDR02 Port: 8001 – Incorrect function.

Resolution:

1. The most common cause of the “Incorrect function” is a physical disk error on the volume or corrupt NTFS filesystem.

Check the system event log for a history or disk/ntfs errors.
Run the chkdsk {volume:} /F /R ( “/R” is important as it will direct chkdsk to scan for bad sectors and attempt to recover)
In cases where disk errors are determined to exist currently or in the past – strongly recommend replacing physical disk
2. Another possible issue is related to the file “AAFailover.md” (used by our drivers) being compressed in the “System Volume Information” directory on the problem volume

Grant everyone permissions to the “System Volume Information” directory on the protected volume
Using Explorer navigate to the “System Volume Information” directory and check the compression attribute of the “AAFailover.md”
Uncompress the “AAFailover.md”
Issue “assuremc disable {volume}” then “assuremc enable {volume}”
Force a new snap
3. The issue that has been showing up repeatedly is Outdated RAID/Storage controller drivers or outdated RAID controller firmware

Outdated RAID or Storage Controller drivers have been shown to interfere with our I/O operations.
The manifestation is – “Incorrect function”
This has been seen now on:
Intel Software RAID ( also called Intel Matrix Storage or later Intel Rapid Storage)
DELL PERC and CERC controllers
RocketRaid from HIghPoint

The solution is to update the storage drivers and controller firmware to the latest provided by manufacturer.
4. “Incorrect function” may be the result of forgetting to reboot an agent after upgrade.

In the future when escalating the “Incorrect function” cases please make sure you verify that none of the issues mentioned above are occurring at customer.
In the case please note:
- Results of chkdsk /f /r
- Whether the RAID controller drivers/Storage controller drivers are the latest available
- State of files in the “System volume information” directory
- Whether you have verified that the agent was rebooted after the upgrade.

Product:

AppAssure 5.x

Problem Description:

When adding an Agent to protection, you get the following error:

Failed to get agent service state at https://<core-name>:8006/apprecovery/api/agent/.
Replay.Common.Contracts.WmiApi.WmiException: Unknown WMI error: Cannot obtain access to remote machine
This error could be caused by one of the following:
- User name or password are incorrect.
- Specified user does not have administrator privileges on the remote machine.
- Specified user does not have access to the remote machine via Windows Management Instrumentation

Resolution:

Try a new administrator username/password to add the agent into protection.
Next, you could log into the Agent system and go into services.msc
Find the AppAssure Agent services and go into properties
Change the logon to a domain admin and restart the services.
Try to re-add back into protection.

Next, try to ping the agent from the core via hostname and then try ip address
If that works, go onto the agent and try to ping the core

You could also try protecting the machine with the IP Address if you are using a host name.

Finally, if all above still fails, please set the following on both the Core and the Agent registry –
-Navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
-Once you are clicked on the key, in the right pane right click and Create new DWORD (Be sure to create the DWORD and NOT the QWORD)
-Name the DWORD: SendTrustedIssuerList
(Note: Be sure to type it exactly as it appears above)
-Once created, right click and modify, make sure that the Decimal Value is 0.

Product:

AppAssure 5.x

Problem Description:

When attempting an installation or upgrade in AppAssure 5.x, the installation fails with ‘Error 1612: The installation source for this product is not available.”

Resolution:

1. Before performing steps below is needed to make a backup for the all AppAssure registry files.
2. Uninstall the existing core installation using a 3rd party removal tool like Msizap.exe
3. stop the Core service
4. Download Ccleaner (http://download.cnet.com/CCleaner/)
5. Delete the ‘Core’ folder from the C:\Program Files\AppRecovery path
6. Run the upgrade
7. Import the registry branch backed up on the first step

Product:

5.x

Problem Description

Backups fail with the following transfer error:
Request to read 0×80 blocks returned a response with 0x8ff54250 bytes, which is longer than the max segment size of 0×400000.

Resolution 1

1. To resolve this issue, make the following modifications:
2. On the agent, Open up Regedit and change the following value:
3. Stop the agent service.
4. HKEY_LOCAL_MACHINE_\SOFTWARE\AppRecovery\Agent\AgentTransferSettings > MetadataAcessTimeout. Change the MetadataAccessTimeout from 1:20 to 5:00.
5. Start the agent service.
6. On Core UI, make time out adjustments for the failing agent (in the Core’s UI-> Agent-> Configuration tab-> Transfer Settings):
7. Maximum concurrent streams. 2
8. Maximum concurrent writes. 2
9. Maximum retries. 3
10. Maximum segment size. 64000
11. Maximum transfer queue depth. 32
12. Outstanding reads per stream. 24
13. Transfer timeout. 0:30:00
14. Snapshot timeout. 0:20:00
15. Network read timeout. 0:10:00
16. Network write timeout. 0:10:00

Resolution 2 for virtual machines

If it ESX(i) machine try to change the NIC to vmxnet3.

http://screencast.com/t/6YpBuUcHAU6

Product:

AppAssure 5.x

Problem Description:

Within AppAssure 5, if the exchange credentials no longer work due to an exchange or windows update of the Core or Agent servers, the Force Checksum Check button can disappear until correct exchange credentials are set.

Resolution:

1. Select the agent.
2. Select Actions > Set Exchange Credentials.
3. Select Actions > Refresh Metadata.
4. Actions > Force > Snapshot.

Once a snapshot is completed with the new credentials the Force checksum check will appear for all recovery points.

Product:

Replay 4.x

Problem Description:

On your Replay SQL Agent, your system event logs are flooding each second with log in failures to an “explicitly specified database” for the Replay service account.

Example:
Login failed for user ‘appassure service account’. Reason: Failed to open the explicitly specified database

Problem Cause:

The cause for this error can be due to a few things, all caused because of problems residing on the locally installed SQL instance.

Resolution:

Verify that when the Replay Agent service is stopped, the events do not populate in the Application Event logs. Afterwards, start the Replay Agent service and again verify that the events begin populating. Open C:\Replay.log (the Replay log file) and look for any failed events accessing a database. You might find the troubled database name entered in the Replay.log.

In the SQL Management Studio, verify that there are no “offline database”. The events in the Application Event logs can appear because of constant attempts to log into these offline databases.

Verify that none of the SQL databases are set for single-user log on restrictions. The events appear in the Application Event log because of the constant attempt to access the database that already has a user logged into it. This can be checked through the steps located in this article: http://technet.microsoft.com/en-us/library/ms345598.aspx

Product:

AppAssure 5.x

Problem Description:

When accessing the Core UI, you may be presented with the following error:
Error
An extension with ‘freeSpaceProgressBar[RepositoryID]‘ name already rendered.
Hide stack trace
An extension with ‘freeSpaceProgressBar[RepositoryID]‘ name already rendered.

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. .

Exception Details: System.Exception: An extension with ‘freeSpaceProgressBar[RepositoryID]‘ name already rendered.

Problem Cause:

This error is a result of the Core UI rendering the data presented within the registry key files. In the instance of this error, you may have recently added a new repository and from the brief glimpses of the UI you get you notice that one of your repositories is missing. The “RepositoryID” is displayed in the error message that is presented and you will need to verify the entries found in the AppRecovery registry key chain.

Resolution:

NOTE: When modifying the registry key, ensure that the Core service has been gracefully stopped and that a backup of the registry is done.
[screenshot]

1) Confirm in the registry key if there are duplicate entries in the “Repositories” branch.
2) If yes, verify which of the keys is accurate. To determine this, verify the paths and extent configurations by dropping down the left carrot of the repository ID.
3) Once you are comfortable with your findings, delete the inconsistent key.
4) Once removed, gracefully start the Core service and see if the Core UI is usable.

Products:

Replay 4 Core Build 4.x

AppAssure 5 Core Build 5.x

Problem Description:

During the snapshot transfer process, the transfer job will fail with the following error:
Error while servicing connection: Error writing 32768 bytes to network – An existing connection was forcibly closed by the remote host.

Problem Cause:
As the snapshot is being transferred, the abrupt termination of the job can be due to multiple reasons. For instance:
1) The repository IO load is too much to handle the acceptance of the snapshot transfer
2) The network connection has timed out or was lost in the middle of the transfer
3) A Replay service was shut down during the middle of the transfer

Resolution:

1) Determine the frequency of the error and how often it appears. See if there is a pattern with this error, such as does it only happen during the late evening hours when nightly tasks take place? The repository may not be able to handle the IO load being requested from snapshot transfers, roll up jobs, and other nightly tasks. Consider what type of disk your repository resides on and if it is suitable for the Replay deployment: http://www.appassure.com/support/KB/repository-options-direct-attached-storage-storage-area-network-or-network-attached-storage/

2) If the errors are frequent and happen very often, consider the network the Replay deployment is using. These timeouts may be beneficial in optimizing the workload for the snapshot transfer: http://www.appassure.com/support/KB/4130354/

3) Verify in the Agent and Core system and application event logs that the Replay service did not unexpectedly terminate itself. In this event logs further detail will help point to mitigation procedures.